Context aware service provision method and apparatus of user device

ABSTRACT

A context aware service provision method and apparatus for recognizing the user context and executing an action corresponding to the user context according to a rule defined by the user and feeding back the execution result to the user interactively are provided. The method for providing a context-aware service includes receiving a user input, the user input being at least one of a text input and a speech input, identifying a rule including a condition and an action corresponding to the condition based on the received user input, activating the rule to detect a context which corresponds to the condition of the rule, and executing, when the context is detected, the action corresponding to the condition.

PRIORITY

This application is a continuation application of prior application Ser.No. 16/054,336, filed on Aug. 3, 2018; which is a continuationapplication of prior application Ser. No. 14/028,021, filed on Sep. 16,2013, which has issued as U.S. Pat. No. 10,042,603 on Aug. 7, 2018; andwhich was based on and claimed priority under 35 U.S.C. 119(a) of aKorean patent application filed on Sep. 20, 2012 in the KoreanIntellectual Property Office and assigned Serial No. 10-2012-0104357,and a Korean patent application filed on Apr. 30, 2013 and assignedSerial No. 10-2013-0048755, the entire disclosures of each of which arehereby incorporated by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a Context Aware Service provisionmethod and apparatus. More particularly, the present invention relatesto a context aware service provision method and apparatus forrecognizing user context and executing an action corresponding to theuser context according to a rule defined by the user and forinteractively feeding back the execution result to the user.

2. Description of the Related Art

With the advance of digital technologies, various types of user devicescapable of communicating and processing data (e.g., a cellularcommunication terminal, a Personal Digital Assistant (PDA), anelectronic organizer, a smartphone, and a tablet Personal Computer (PC))are emerging. Recently, user devices are evolving into multifunctionaldevices integrating various functions in line with the mobileconvergence tendency. For example, a recent user device integratesvarious functions including a voice and video telephony function, amessaging function including Short Message Service/Multimedia MessageService (SMS/MMS) and email, a navigation function, a picture capturingfunction, a broadcast playback function, a multimedia (e.g., video andaudio) playback function, an Internet access function, a messengerfunction, a Social Networking Service (SMS) function, and the like.

Meanwhile, there is a growing interest in Context Aware Service (CAS)using various life-log technologies that are developed to record thedaily life of an individual in the form of digital information. The CASis characterized in that determination as to whether to provide theservice and determination as to service content to be provided are madeaccording to the change of the context defined by a service object. Theterm ‘Context’ means the information for use in determining the servicebehavior defined by the service object, the information includingservice provision timing, whether to provide service, target to serve,service provision location, and the like. Such technologies are capableof recording various types of information characterizing the behavior ofan individual and providing the CAS based on the recorded information.

However, the CAS method according to the related art is implementedunder the assumption of burdensome installation of various sensordevices based on a domain for collecting information on the individual.The CAS system according to the related art consists of user devices forcollecting data by means of sensors and a server for analyzing the dataacquired from the user devices to build a context and executing theservice based on the context. For example, because the user device hasto be equipped with various sensors and must interoperate with theserver to provide the user context-based service, the high systemimplementation cost and design complexity are obstacles for implementingthe CAS system according to the related art.

The CAS system according to the related art has a drawback associatedwith the difficulty in providing the context-based service efficientlydue to the limit to the information collected via the user device andlack of effective learning process. For example, the CAS systemaccording to the related art can provide the user with the context-basedservice only with the rule defined by the device manufacturer so as notto satisfy the requirements of all users. The CAS system according tothe related art has a drawback of low user accessibility because theuser has to execute an extra program and/or carry out complexmanipulation to use the context-based service. Furthermore, the CASsystem according to the related art is limited to a single context awarescenario so as to provide no flexibility in setting conditions by takingnotice of various situations.

Therefore, a need exists for a CAS method and apparatus that is capableof supporting CAS with one or more rules defined by the user.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present invention.

SUMMARY OF THE INVENTION

Aspects of the present invention are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentinvention is to provide a Context Aware Service (CAS) method andapparatus that is capable of supporting a CAS with one or more rulesdefined by the user.

Another aspect of the present invention is to provide a CAS method andapparatus that is capable of feeding back the context informationcollected based on one or more rules to the user in such a way of being,at the terminal, aware of the situation of the user which is determinedaccording to the rules predefined by the user and executing the actioncorresponding to the user's situation.

Another aspect of the present invention is to provide a CAS method andapparatus that is capable of allowing the user to define rules (orsituations), commands for executing the rules, and actions to beexecuted based on the rules by inputting natural language-based textand/or voice to the user device.

Another aspect of the present invention is to provide a CAS method andapparatus that is capable of expending CAS supportability in such a wayof defining, at the user device, rules, commands, and actions usingnatural language-based text or voice, recognizing the naturallanguage-based text or voice, and executing the rule selected accordingto the motion of user device.

Another aspect of the present invention is to provide a CAS method andapparatus that is capable of configuring plural conditions of a rule,being aware of multiple contexts corresponding to the respectiveconditions, and execution multiple actions corresponding to therespective contexts.

Another aspect of the present invention is to provide a CAS method andapparatus that is capable of configuring one or more conditionsaccording to the user's preference in defining a rule.

Another aspect of the present invention is to provide CAS method andapparatus that is capable of improving user convenience and deviceusability with the implementation of optimal CAS environment.

In accordance with an aspect of the present invention, a method forproviding a context-aware service of a user device is provided. Thecontext-aware service provision method includes receiving a user input,the user input being at least one of a text input and a speech input,identifying a rule including a condition and an action corresponding tothe condition based on the received user input, the user input being oneof a text input and a speech input, activating the rule to detect acontext which corresponds to the condition of the rule, and executing,when the context is detected, the action corresponding to the condition.

In accordance with another aspect of the present invention, a method forproviding a context-aware service of a user device is provided. Thecontext-aware service provision method includes providing a userinterface for configuring a rule, receiving at least one of a naturallanguage-based speech input and a natural language-based text inputthrough the user interface, configuring the rule with a condition and anaction identified from the user input, activating the rule to detect anevent which corresponds to the condition of the rule, and executing,when the event is detected, the action corresponding to the condition.

In accordance with another aspect of the present invention, a method forproviding a context-aware service of a user device is provided. Themethod includes receiving a user input for configuring a rule withnatural language-based voice or text, configuring the rule according tothe user input, receiving a command for activating the rule, the commandbeing one of a natural language-based speech, a natural language-basedtext, a motion detection event of the user device, a receipt of inboundsound, and a receipt of inbound message, executing the rulecorresponding to the command, checking at least one condition occurringinside and outside as specified in the rule, and executing, when atleast one condition specified in the rule is fulfilled, at least oneaction corresponding to at least one of the at least one fulfilledcondition.

In accordance with another aspect of the present invention, a method forproviding a context-aware service of a user device is provided. Thecontext-aware service provision method includes defining a rule,receiving a command input for executing the rule, executing the rule inresponse to the command, checking a condition corresponding to the rule,executing at least one action when the condition corresponding to therule is detected.

In accordance with another aspect of the present invention, acontext-aware service provision method of a user device is provided. Thecontext-aware service provision method includes monitoring to detectwhether an event occurs in the state in which a rule is executed,extracting, when an event is detected, a function specified to executean action, executing the action according to the function, feeding backinformation related to the execution of the action, determining, when noevent is detected, whether a current situation fulfills a rule releasecondition, and releasing, when the current situation fulfills a rulerelease condition, the rule.

In accordance with another aspect of the present invention, anon-transient computer-readable storage medium stores a program ofexecuting the above method by means of a processor.

In accordance with another aspect of the present invention, a userdevice is provided. The user device includes a storage unit which storesa rule including a condition and an action corresponding to thecondition, a display unit which displays a user interface for receivinga user input and execution information in a state in which the rule isactivated and an execution result of the action, and a control unitwhich controls identifying the rule including the condition and theaction based on the user input, the user input being at least one of atext input and a speech input, activating the rule to detect a contextwhich corresponds to the condition of the rule, and executing, whencontext is detected, the a action corresponding to the condition.

In accordance with another aspect of the present invention, a userdevice is provided. The user device includes a rule configuration modulewhich is computer-implemented for receiving a user input and foridentifying a rule including a condition and an action corresponding tothe condition based on the user input, the user input being at least oneof a natural language-based speech input and a natural language-basedtext input, a rule execution module which is computer-implemented forreceiving a command for activating the rule, the command being one of anatural language-based speech, a natural language-based text, a motiondetection event of the user device, a receipt of inbound sound, and areceipt of an inbound message, and for executing the rule correspondingto the command, a condition check module which is computer-implementedfor detecting a context which corresponds to the condition as specifiedin the rule, and an action execution module which iscomputer-implemented for executing, when the context is detected, theaction corresponding to the condition.

In accordance with another aspect of the present invention, anon-transient computer-readable storage medium is provided. Thenon-transient computer-readable storage medium includes a program that,when executed, causes at least one processor to perform a methodincluding defining a rule for context-aware service according to userinput, executing, when a command for executing the rule is received, therule corresponding to the command, and executing, when a conditionspecified in the rule is fulfilled, an action corresponding to thecondition.

In accordance with still another aspect of the present invention, anon-transient computer-readable storage medium is provided. Thecomputer-readable storage medium includes a program that, when executed,causes at least one processor to perform a method including receiving auser input, the user input being at least one of a text input and aspeech input, identifying a rule including a condition and an actioncorresponding to the condition based on the received user input,activating the rule to detect a context which corresponds to thecondition of the rule, and executing, when the context is detected, theaction corresponding to the condition.

Other aspects, advantages, and salient features of the invention willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainexemplary embodiments of the present invention will be more apparentfrom the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram illustrating a configuration of a user deviceaccording to an exemplary embodiment of the present invention;

FIG. 2 is a flowchart illustrating a Context Aware Service (CAS)provision method of a user device according to an exemplary embodimentof the present invention;

FIGS. 3A to 3K are diagrams illustrating exemplary screens forexplaining an operation of generating a rule in a user device accordingto an exemplary embodiment of the present invention;

FIGS. 4A to 4J are diagrams illustrating exemplary screens forexplaining an operation of generating a rule in a user device accordingto an exemplary embodiment of the present invention;

FIGS. 5A to 5E are diagrams illustrating exemplary screens forexplaining an operation of executing a predefined rule in a user deviceaccording to an exemplary embodiment of the present invention;

FIG. 6 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention;

FIG. 7 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention;

FIG. 8 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention;

FIG. 9 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention;

FIG. 10 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention;

FIG. 11 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention;

FIG. 12 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention;

FIG. 13 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention;

FIGS. 14A and 14B are diagrams illustrating exemplary screens forexplaining operations of stopping a currently running rule temporarilyin a user device according to an exemplary embodiment of the presentinvention;

FIGS. 15A and 15B are diagrams illustrating exemplary screens forexplaining operations of stopping a currently running rule temporarilyin a user device according to an exemplary embodiment of the presentinvention;

FIGS. 16A to 16C are diagrams illustrating exemplary screens forexplaining operations of stopping currently running rule in a userdevice according to an exemplary embodiment of the present invention;

FIG. 17 is a diagram illustrating an exemplary screen with an indicationof an execution rule in a user device according to an exemplaryembodiment of the present invention;

FIGS. 18A and 18B are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention;

FIGS. 19A and 19B are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention;

FIGS. 20A to 20C are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention;

FIGS. 21A and 21B are diagrams illustrating exemplary screens associatedwith an operation of notifying of an execution rule in a user deviceaccording to an exemplary embodiment of the present invention;

FIGS. 22A to 22C are diagrams illustrating exemplary screens associatedwith an operation of terminating a currently running rule in a userdevice according to an exemplary embodiment of the present invention;

FIGS. 23A and 23B are diagrams illustrating exemplary screens associatedwith an operation of terminating a currently running rule in a userdevice according to an exemplary embodiment of the present invention;

FIGS. 24 and 25 are diagrams illustrating a situation of terminating CASservice in a user device according to an exemplary embodiment of thepresent invention;

FIGS. 26A and 26B are diagrams illustrating exemplary screens associatedwith an operation of deleting a rule in a user device according to anexemplary embodiment of the present invention;

FIG. 27 is a flowchart illustrating a procedure of generating a rule ina user device according to an exemplary embodiment of the presentinvention;

FIG. 28 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention;

FIG. 29 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention;

FIG. 30 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention;

FIGS. 31A to 31N are diagrams illustrating exemplary screens associatedwith an operation of generating a rule in a user device according to anexemplary embodiment of the present invention;

FIGS. 32A to 32E are diagrams illustrating exemplary screens associatedwith an operation of executing a rule in a user device according to anexemplary embodiment of the present invention;

FIGS. 33A to 33D are diagrams illustrating exemplary screens associatedwith an operation of suspending a currently running rule in a userdevice according to an exemplary embodiment of the present invention;and

FIGS. 34A to 34D are diagrams illustrating exemplary screens associatedwith an operation of terminating a currently running rule in a userdevice according to an exemplary embodiment of the present invention.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, description of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of theinvention. Accordingly, it should be apparent to those skilled in theart that the following description of exemplary embodiments of thepresent invention is provided for illustration purpose only and not forthe purpose of limiting the invention as defined by the appended claimsand their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

Exemplary embodiments of the present invention relate to a Context AwareService (CAS) provision method and apparatus of a user device.

According to an exemplary embodiment of the present invention, the userdevice is capable of being aware of distinct contexts of the useraccording to one or more rules defined by the user.

According to an exemplary embodiment of the present invention, the userdevice is capable of executing one or more actions according to thecontext awareness and, as a consequence of the executed action, feedingback the context information to the user or a predetermined person.

According to an exemplary embodiment of the present invention, the CASprovision method and apparatus is capable of feeding back the contextinformation to the user by means of an external device (e.g., atelevision, an electric lamp, and the like) and/or transmitting thecontext information to another user through a message.

In various exemplary embodiments of the present invention, a rule can bedefined through text (e.g., hand writing) or voice input using a naturallanguage. In various exemplary embodiments of the present invention, thenatural language corresponds to the language used by humans as comparedto an artificial language (or machine language) invented for a certainpurpose.

In various embodiments of the present invention, a rule may be activatedin response to input (or receipt of) a command associated with the rule.

In various embodiments of the present invention, a rule may beidentified or selected based on a received user input. The identifiedrule may be activated to detect a context which corresponds to thecondition of the rule. When the context is detected, the actioncorresponding to the condition may be executed.

In various exemplary embodiments of the present invention, when a ruleis activated, the user device may monitor or detect the context withinwhich the user device operates. Based on the monitored or detectedcontext within which the user device operates, the user device maydetermine or identify that the user device operates in a contextcorresponding to an activated rule.

In various exemplary embodiments of the present invention, the rule canbe composed of at least one condition and at least one action, and amethod or process for generating the rule is described hereafter.

In various exemplary embodiments of the present invention, apredetermined rule can be executed in response to the receipt of aninstruction corresponding to the rule.

In various exemplary embodiments of the present invention, theinstruction may include a natural language-based text or voice commandor command sentence input by means of various input means (e.g., atouchscreen, a keypad, a microphone, and the like). The instruction alsomay include a change in the user device (e.g. a change in posture,orientation, or the like) which is detected by various sensors of theuser device (e.g., a proximity sensor, a luminance sensor, anacceleration sensor, a gyro sensor, a voice sensor, and the like)according to a predetermined rule. The instruction also may include areceipt of an inbound message or inbound sound corresponding to apredetermined rule. The instruction also may include the change ingeographical location of the user (or user device) in correspondence toa predetermined rule.

In various exemplary embodiments of the present invention, theinstruction (e.g., a definition on command, a command sentence, asensible behavior of user device, a sensor for sensing the behavior, andthe like) for executing the rule can be configured in such a way ofentering natural language-based voice or text.

In various exemplary embodiments of the present invention, the commandor command sentence as one type of the instruction executing the rulecan be entered in the form of a part (e.g., word), partial sentence orcomplete sentence of the natural language that is included in definingthe rule.

In various exemplary embodiments of the present invention, the sentencecan be a minimum expression unit for a complete content in expressing athought or emotion and, although it may be essential to include asubject and a predicate, any of the subject and predicate may beomitted.

In the various exemplary embodiment of the present invention, thedetection of user device's behavior as one type of the instruction canbe input by the operation of one or more sensors configured in definingthe rule.

In various exemplary embodiments of the present invention, the action iscapable of including the operation executed by the user device uponawareness of the situation specified in the currently running rule.

In various exemplary embodiments of the present invention, the action iscapable of including the operation control (e.g., internal operationcontrol) for feedback of the information on the situation specified inthe corresponding rule by controlling the internal components (e.g.,display unit, communication module, speaker), the operation control(e.g., external operation control) for feedback the information on thesituation specified in the corresponding rule by controlling an externalcomponents (e.g., television, electric lamp, external speaker), and theoperation control for controlling both the internal and externalcomponents of the user device.

In various exemplary embodiments of the present invention, CAS denotesthe service in which the user device is aware of the situation specifiedin the rule defined by the user, executes an action corresponding to thesituation, and provides the user (or predetermined person) with theinformation on the situation as a consequence of the execution of theaction. The situation information includes all information available atthe time of user interaction such as user (or user device) location,identifier, activity, state, and application of the user device.

The configuration and operation control method of the user deviceaccording to exemplary embodiments of the present invention is describedhereinafter with reference to the accompanying drawings. It is notedthat exemplary embodiments of the present invention are not limited tothe configuration and operation control method of the user deviceaccording to the following description but can be implemented withvarious changes and modifications without departing from the scope ofthe present invention.

FIG. 1 is a block diagram illustrating the configuration of a userdevice according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the user device 100 includes a radio communicationunit 110, an input unit 120, a touchscreen 130, an audio processing unit140, a storage unit 150, an interface unit 160, a control unit 170, anda power supply 180. The user device 100 according to an exemplaryembodiment of the present invention can be implemented with or withoutat least one of the components depicted and non-depicted in FIG. 1. Forexample, if the user device 100 according to an exemplary embodiment ofthe present invention supports a picture capture function, then a cameramodule (not shown) can be included. Similarly, if the user device 100has no broadcast reception and playback function, some function modules(e.g., broadcast reception module 119 of the radio communication unit110) may be omitted.

The radio communication unit 110 may include at least one module capableof radio communication of the user device 100 with a radio communicationor another device. For example, the radio communication unit 110 mayinclude a cellular communication module 111, a Wireless Local AreaNetwork (WLAN) module 113, a short range communication module 115, alocation positioning module 117, and a broadcast reception module 119.

The cellular communication module 111 is capable of communicating radiosignals with at least one of a base station of a cellular communicationnetwork, an external device, and various servers (e.g., an integrationserver, a provider server, a content server, an Internet server, a cloudserver). The radio signals may carry the voice telephony data, videotelephony data, text/multimedia message data, and the like. The cellularcommunication module 111 is capable of connecting to a provider serveror a content server to download various rules for CAS under the controlof the control unit 170. The cellular communication module 111 iscapable of transmitting the action execution result (e.g., situationinformation) to at least one target user device 100 under the control ofthe control unit 170. The cellular communication module 111 is alsocapable of receiving the message generated when the condition defined inor otherwise associated with a currently running rule is fulfilled(e.g., satisfied) under the control of the control unit 170.

The WLAN module 113 is responsible of establishing a WLAN link with anAccess Point (AP) or another user device 100 and capable of beingembedded in the user device 100 or implemented as an external device.There are various radio Internet access technologies available such asWi-Fi, Wireless Broadband (WiBro), World Interoperability for MicrowaveAccess (WiMAX), and High Speed Downlink Packet Access (HSDPA), and thelike. The WLAN module 113 is capable of receiving various types of data(e.g., including rules) for CAS in the state connected to a server. TheWLAN module 113 is capable of transmitting and receiving various data(e.g., including rules) to and from another user device according to theuser's intention in the state in which a WLAN link has been establishedwith another user device. The WLAN module 113 is also capable oftransmitting and receiving various data (e.g., including rules) for CASto and from a cloud server through a WLAN link. The WLAN module 113 isalso capable of transmitting the action execution result (e.g.,situation information) to at least one target user device under thecontrol of the control unit 170. The WLAN module 113 is also capable ofreceiving a message generated when a condition specified in thecurrently running rule is fulfilled under the control of the controlunit 170.

The short range communication module 115 is responsible for the shortrange communication of the user device 100. There are various shortrange communication technologies available such as Bluetooth, BluetoothLow Energy (BLE), Radio Frequency Identification (RFID), Infrared DataAssociation (IrDA), Ultra Wideband (UWB), ZigBee, and Near FieldCommunication (NFC), and the like. The short range communication module115 is capable of transmitting and receiving various data (includingrules) for CAS to and from another user device according to the user'sintention when the user device 100 is connected to the another userdevice.

The location positioning module 117 is responsible for positioning thelocation of the user device 100. The location positioning module 117 mayinclude a Global Positioning System (GPS) module, and/or the like. Thelocation positioning module 117 collects accurate distance informationand time information from at least three base stations and performstriangulation based on the acquired information to acquire 3-Dimensional(3D) location information with latitude, longitude, and altitude. Thelocation positioning module 117 is also capable of calculating thelocation information based on the signals from three or more satellitesin real time. The location information of the user device 100 can beacquired using various methods.

The broadcast reception module 119 receives broadcast signals (e.g., TVbroadcast signal, radio broadcast signal, and data broadcast signal)and/or information on the broadcast (e.g., broadcast channelinformation, broadcast program information, and broadcast serviceprovider information) from an external broadcast management serverthrough a broadcast channel (e.g., satellite broadcast channel andterrestrial broadcast channel).

The input unit 120 generates an input signal for controlling theoperation of the user device 100 in response to the user input. Theinput unit 120 may include a key pad, a dome switch, a touch pad (e.g.,using a capacitive technology, a resistive technology, and the like), ajog wheel, a jog switch, a sensor (e.g., voice sensor, proximity sensor,luminance sensor, acceleration sensor, and gyro sensor), and the like.The input unit 120 can be implemented with external buttons and/orvirtual button on the touch panel. The input unit 120 is capable ofgenerating an input signal in response to the user input (e.g., a textinput, a voice input, a user device motion input, and the like) fordefining or executing the rule (e.g. an instruction).

The touchscreen 130 is an input/output means responsible for input andoutput functions simultaneously and includes a display panel 131 and atouch panel 133. According to an exemplary embodiment of the presentinvention, if a user's touch gesture (e.g., one or more touches, a tap,a drag, a sweep, a flick, and the like) is detected by the touch panelin the state in which an execution screen of the user device 100 (e.g.,a rule (condition and action) configuration screen, an outboundcall-placing screen, a messenger screen, a game screen, a galleryscreen, and the like) is displayed on the display panel 131, thetouchscreen 130 generates an input signal corresponding to the touchgesture to the control unit 170. The control unit 170 identifies thetouch gesture and executes an operation according to the touch gesture.For example, if a touch gesture of natural language-based text writingis detected on the touch panel 133 in the state in which the ruleconfiguration screen is displayed on the display panel 131, the controlunit 170 generates a rule in response to the touch gesture.

The display panel 131 displays (outputs) the information processed bythe user device 100. For example, if the user device is operating in atelephony mode, the display unit 131 displays a telephony User Interface(UI) or Graphic UI (GUI). If the user device 100 is operating in a videotelephony mode or a picture capture mode, the display unit 131 displaysa UI or GUI displaying the picture captured by the camera or receivedthrough the communication channel. According to exemplary embodiments ofthe present invention, the display panel 131 is capable of displaying aUI or a GUI related to the CAS operation. For example, the display panel131 is capable of providing the various UIs or GUIs displaying the ruleconfiguration and rule execution states in response to a user input, anaction execution state, and an action execution result (e.g., situationinformation). The display panel 131 is also capable of supportingdisplay mode switching function for switching between portrait mode andlandscape mode according to the rotation direction (or orientation) ofthe user device 100. The operation of the display panel 131 is describedlater with reference to exemplary screens.

The display panel 131 can be implemented any of a Liquid Crystal Display(LCD), a Thin Film Transistor LCD (TFT LCD), a Light Emitting Diode(LED), an Organic LED (OLED), an Active Matrix OLED (AMOLED), a flexibledisplay, a bended display, a 3-Dimensional (3D) display, and the like.The display panel 131 can be implemented as a transparent orsemitransparent display panel through which the light penetrates.

The touch panel 133 is capable of being placed on the display panel 131to detect the user's touch gesture made on the surface of the touchscreen 130 (e.g., a single touch gesture, and a multi-touch gesture). Ifthe user's touch gesture is detected on the surface of the touchscreen130, the touch panel 133 extracts the coordinates at the position of thetouch gesture and transfers the coordinates to the control unit 170. Thetouch panel 133 detects the touch gesture made by the user and generatesa signal corresponding to the touch gesture to the control unit 170. Thecontrol unit 170 is capable of executing a function according to thesignal transmitted by the touch panel 133 in association with theposition at which the touch gesture is detected.

The tough panel 133 can be configured to convert the pressure applied ata specific position of the display panel 131 or the change ofcapacitance at a specific position of the display panel 131 to anelectrical input signal. The touch panel 133 is capable of measuring thepressure of the touch input as well as the position and size of thetouch. If a touch input is detected, the touch panel 133 generatescorresponding signal(s) to a touch controller (not shown). The touchcontroller (not shown) is capable of processing the signal(s) andtransferring the corresponding data to the control unit 170. In thisway, the control unit 170 is capable of determining the touched area onthe display panel 131.

The audio processing unit 140 sends the audio signal received from thecontrol unit 170 to the speaker (SPK) 141 and sends the audio signalsuch as voice input through the microphone (MIC) 143 to the control unit170. The audio processing unit 140 is capable of processing thevoice/sound data to output an audible sound wave through the speaker 141and capable of processing the audio signal including voice to generate adigital signal to the control unit 170.

The speaker 141 is capable of outputting audio received by the radiocommunication unit 110 or stored in the storage unit 150 in thetelephony mode, audio (video) recording mode, speech recognition mode,broadcast reception mode, photo shooting mode, and CAS mode The speaker141 is also capable of outputting sound effects associated with thefunction executed in the user device 100 (e.g., rule execution, actionexecution, context information feedback, inbound call reception,outbound call placing, photo shooting, and media content (e.g., audioand video) playback).

The microphone 143 is capable of processing the input acoustic signal togenerate voice data in the telephony mode, audio (video) recording mode,speech recognition mode, broadcast reception mode, photo shooting mode,CAS mode, and the like. The processed voice data can be processed intothe signal to be transmitted to the base station by means of thecellular communication module 111 in the telephony mode. The microphone143 can be implemented with various noise cancellation algorithms tocancel the noise occurring in the audio signal input process. Themicrophone 143 is capable of processing the user input (e.g., naturallanguage-based text or voice input) for rule definition and execution(instruction) to generate the corresponding input signal to the controlunit 170.

The storage unit 150 is capable of storing the programs associated withthe processing and control operations of the control unit 170 andtemporarily saving the input/output data (e.g., rule information,instruction information, action information, context information,contact information, message, media content (e.g. audio, video, ande-book), and the like). The storage unit 150 is also capable of storinguse frequencies of user device functions (e.g., rule use frequency,instruction use frequency, application use frequency, data (e.g. phonenumber, message, and media content), importance rate, priority, and thelike). The storage unit 150 is also capable of storing the data relatedto the vibration patterns and sound effects output in response to theuser input made by means of the input unit 120 and/or the touchscreen130. According to exemplary embodiments of the present invention, thestorage unit 150 is capable of storing a mapping table including themappings among the instructions per the user-defined rule and actions(e.g., functions and applications) per rule, and rule terminationconditions.

The storage unit 150 stores Operating System (OS) of the user device100; programs associated with input and display control operations ofthe touchscreen 130, CAS control operations (e.g., rule includingconditions and actions) of context awareness depending on the rule(e.g., condition), action execution according to the rule, and contextinformation feedback; and data generated by the programssemi-persistently or temporarily. According to exemplary embodiments ofthe present invention, the storage unit 150 is also capable of storingsetting information for supporting CAS. The setting information iscapable of including the information on whether to support voice-basedCAS or text-based CAS. The setting information is also capable ofincluding at least one condition per rule and the rules specifying theactions corresponding to the conditions.

The storage unit 150 can be implemented with a storage medium of atleast one of a flash memory type, a hard disk type, a micro type, a cardtype (e.g., a Secure Digital (SD) type and an eXtreme Digital (XD) cardtype) memories, Random Access Memory (RAM), Dynamic RAM (DRAM), StaticRAM (SRAM), Read-Only Memory (ROM), Programmable ROM (PROM),Electrically Erasable PROM (EEPROM), Magnetic RAM (MRAM), a magneticdisk, an optical disk type memories, and the like. The user device 100can interoperate with a web storage working as the storage unit 150 onthe Internet.

The interface unit 160 provides the interface for the external devicesconnectable to the user device 100. The interface unit 160 is capable oftransferring the data or power from the external devices to the internalcomponents of the user device 100 and transferring the internal data tothe external devices. For example, the interface unit 160 can beprovided with wired/wireless headset port, external charging port,wired/wireless data port, memory card slot, identity module slot, audioinput/output port, video input/output port, earphone jack, and the like.

The control unit 170 controls overall operations of the user device 100.For example, the control unit 170 is capable of controlling the voicetelephony, data telephony, and video telephony functions. In variousexemplary embodiments of the present invention, the control unit 170 isalso capable of controlling the operations associated with CAS. Invarious exemplary embodiments of the present invention, the control unit170 may include a data processing module 171 having a rule configurationmodule 173, a rule execution module 175, a condition check module 177,and an action execution module 179. The operations of the ruleconfiguration module 173, rule execution module 175, condition checkmodule 177, action execution module 179 are described later withreference to the accompanying drawings. The control unit 170 may includea CAS framework (not shown) for supporting CAS and a multimedia module(not shown) for multimedia playback. In an exemplary embodiment of thepresent invention, the CAS framework (not shown) and the multimediamodule (not shown) can be embedded in the control unit 170 orimplemented as an independent module.

According to an exemplary embodiment of the present invention, thecontrol unit 170 is capable of controlling CAS-related operations suchas user-defined rule configuration, rule-based context awareness,rule-based action execution, and context information feedback as aconsequence of the action execution. The control unit 170 (e.g., therule configuration module 173) is capable of defining the rule forproviding CAS according to the user input (e.g. natural language-basedspeech or text input). The control unit 170 is capable of operativelyreceiving a user input, and identifying a rule including a condition andan action corresponding to the condition based on the received userinput. The control unit 170 is capable of activating the rule to detecta context which corresponds to the condition of the rule. If aninstruction for executing the rule which is specified in the configuredrule is detected, the control unit 170 (e.g., the rule execution module175) is capable of executing one or more rules. The control unit 170(e.g., the condition check module 177) is capable of checking (e.g.,determining) and recognizing the condition (or context) as a consequenceof the rule execution. If the condition specified in the correspondingrule is recognized, the control unit 170 (e.g., the action executionmodule 179) executes the action triggered when the condition isfulfilled. For example, the control unit 170 executes the actioncorresponding to the condition when the context is detected. The controlunit 170 (e.g., the action execution module 179) executes at least onefunction (application) to perform the operation corresponding to thefunction (or application).

If an event (action fulfilling the condition specified in the rule) isdetected by the condition check module 177 in the state in which atleast one rule is executed by the rule execution module 175 in responseto the user request, the control unit 170 extracts the function definedfor executing an action corresponding to the event in the currentlyrunning rule by means of the action execution module 179. The controlunit 170 is capable of controlling the execution of the actioncorresponding to the function extracted by the action execution module179. If no user request event is detected, the control unit 170determines whether the current situation fulfills the condition forterminating at least one currently running rules. If the currentsituation fulfills the condition, the control unit 170 controls the ruleexecution module 175 to terminate the currently running rules.

The control unit 170 is also capable of controlling the operation offeedback of the context information as a consequence of the actionexecution by the action execution module 179 when the actioncorresponding to the condition checked (e.g., determined) by thecondition check module 177 is executed in the state in which at leastone rule is executed by the rule execution module 175. The control unitis also capable of controlling the feedback operation in correspondenceto the action termination by the action execution module 179 when therule executed according to the condition checked (e.g., determined) bythe condition check module 177 is terminated in the state in which atleast one rule is executed by the rule execution module 175.

In various exemplary embodiments of the present invention, the feedbackoperation according to the action execution may include presenting theaction execution result (e.g., context information) to the user throughthe display panel 131 and transmitting the action execution result(e.g., context information) to another user by means of the radiocommunication unit 110. The feedback operation according to the actionexecution may include transmitting to the corresponding external devicea control signal for controlling the operation (e.g., turn-on/off) ofthe external device (e.g., a light lamp, a television, and the like) incorrespondence to the action execution.

In various exemplary embodiments of the present invention, the feedbackoperation may include providing the device user with at least one of anaudio effect (e.g., predetermined sound effect through the speaker 141),a visual effect (e.g., predetermined screen through the display panel131), and a haptic effect (e.g., predetermined vibration pattern througha vibration module (not shown)).

The detailed control operations of the control unit 170 will be moreclarified in the description on the operation and control method of theuser device 100 which is made later with reference to the accompanyingdrawings.

In various exemplary embodiments of the present invention, the controlunit 170 is capable of controlling the operations related to the normalfunctions of the user device 100 as well as the above describedoperations. For example, the control unit 170 is capable of controllingthe execution of an application and the display of the execution screen.The control unit 170 is also capable of controlling the operations ofreceiving the input signal generated by the touch-based input interface(e.g., touchscreen 130) in response to a touch gesture and executing thefunction according to the input signal. The control unit 170 is alsocapable of communicating various data through wired or wirelesschannels.

The power supply 180 supplies the power from an external power source oran internal power source to the internal components of the user device100.

As described above, according to exemplary embodiments of the presentinvention, the user device 100 includes a rule configuration module 173for configuring a computer-executable rule in response to a naturallanguage-based user's voice or text input for configuring a rule, a ruleexecution module 175 for executing the computer-executable rule inresponse to the instruction for executing the rule in the form of anatural language-based voice or text instruction or user device'smotion-responsive instruction or a message instruction from outside, acondition check module 177 for checking (e.g., recognizing and/ordetermining) whether at least one condition (e.g., situation) specifiedin the rule is fulfilled, and an action execution module 179 forexecuting at least one action depending on whether the conditionspecified in the rule is fulfilled.

In various exemplary embodiments of the present invention, the ruleconfiguration module 175 is capable of operating to be aware of thenatural language-based voice or text input made by the user in the ruleconfiguration mode. In various exemplary embodiments of the presentinvention, the rule configuration module 175 is capable of configuringplural conditions per rule and mapping plural actions to the conditions.In various exemplary embodiments of the present invention, the conditioncheck module 177 is capable of performing multi-context awarenessfunction for checking the plural contexts corresponding to theconditions configured per rule. In various exemplary embodiments of thepresent invention, the action execution module 179 is capable ofperforming the plural actions simultaneously or sequentially in responseto the awareness of multiple contexts of a rule.

The CAS provision method according to one of various exemplaryembodiments of the present invention can be implemented in software,hardware, or a combination of both and stored in a non-transientcomputer-readable storage medium. In the case of the hardwareimplementation, the CAS provision method according to exemplaryembodiments of the present invention can be implemented with at leastone of Application Specific Integrated Circuits (ASICs), Digital SignalProcessors (DSPs), Digital Signal Processing Devices (DSPDs),Programmable Logic Devices (PLDs), Field Programmable Gate Arrays(FPGAs), processors, controllers, micro-controllers, microprocessors,and other electrical units which perform certain tasks.

Exemplary embodiments of the present invention can be implemented by thecontrol unit 170 itself. In the case of being implemented in software,the procedures and functions described in the exemplary embodiments ofthe present invention can be implemented with the software modules(e.g., rule configuration module 173, rule execution module 175,condition check module 177, and action execution module 179, and thelike). The software modules are capable of performing at least one ofabove-described functions and operations described.

The storage medium can be any of a non-transient computer-readablestorage media storing the program commands of defining a rule for CAS inresponse to the user input, executing at least one rule in response to arule execution instruction, and executing at least one actioncorresponding to a condition specified in the execution rule when acondition (situation) specified in the rule is fulfilled. The storagemedium also can be a non-transient computer-readable storage mediumstoring the program commands of configuring a rule including conditionsand actions corresponding to the conditions in response to the naturallanguage-based voice or text input made by the user, activating the rulein response to an instruction indicating the rule, determining whetherthe condition specified in the executed rule is fulfilled, and executingthe action corresponding to the condition fulfilled.

In various exemplary embodiments of the present invention, the userdevice 100 can be any type of information communication devices,multimedia devices, and their equivalents having any of ApplicationProcessor (AP), Graphic Processing Unit (GPU), and Central ProcessingUnit (CPU). For example, the user device 100 can be any of a cellularcommunication terminal operating with various communication protocolscorresponding to the communication systems, a tablet Personal Computer(PC), a smartphone, a digital camera, a Portable Multimedia Player(PMP), a Media Player (e.g., an MP3 player), a portable game console, aPersonal Digital Assistant (PDA), and the like. The CAS provision methodaccording to any of various exemplary embodiments of the presentinvention also can be applied to various display devices such as adigital television (TV), a Digital Signage (DS), a Large Format Display(LFD), a laptop computer, a desktop computer, and the like.

FIG. 2 is a flowchart illustrating a CAS provision method of a userdevice according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the control unit 170 (e.g., rule configurationmodule 173) defines (e.g., configures and generates) a rule at step 201in response to a user input for defining the rule for CAS by means ofone of the input unit 120, the microphone 143, and touchscreen 130.

For example, the user is capable of inputting a natural language-basedspeech for configuring the rule by means of the microphone 143 in therule configuration mode. The user is also capable of inputting a naturallanguage-based text for configuring the rule by means of the touchscreen130 in the rule configuration mode. The control unit 170 (e.g., the ruleconfiguration module 173) recognizes and parses the user input (e.g.,speech recognition and text recognition) to define (e.g., identify) therule to be executed. The control unit 170 (e.g., the rule executionmodule 175) may control the user device 100 to enter the activationstate and waits for the execution of the rule configured in response tothe user input (e.g., instruction for executing the rule). The ruleconfiguration and generation operations according to various exemplaryembodiments of the present invention are described with reference to theaccompany drawings (e.g., FIGS. 3A to 3K and 31A to 31N).

If an instruction for executing a specific rule is received in the statein which at least one rule is defined in response to the user input, thecontrol unit 170 (e.g., the rule execution module 175) controls toexecute the corresponding rule at step 203.

For example, the user is capable of inputting a natural language-basedcommand or command sentence for executing the predefined rule by meansof one of the input unit 120, the microphone 143, and the touchscreen130. The user is capable of executing the corresponding rule by enteringa specific instruction targeting at least one rule for activating CASusing a function key input, a voice input, a touch input (e.g., textwriting and selecting widget), and a gesture-based input (e.g., changingthe posture of the user device 100 such as tilting and accelerationmovement). According to exemplary embodiments of the present invention,the instruction for executing the rule can be generated in varioususer's input behaviors fulfilling the conditions specified in thecorresponding rule. According to exemplary embodiments of the presentinvention, the instruction for executing the rule can be generated inthe form of receiving a specific message or sound fulfilling a conditionspecified in the rule. The control unit 170 (e.g., the rule executionmodule 175) is capable of recognizing the instruction fulfilling thecondition for executing the rule and executing the corresponding rule toactivate the CAS in response to the recognized instruction.

The control unit 170 (e.g., the condition check module 177) triggers thecondition (situation) corresponding to the currently running rule atstep 205.

If the condition corresponding to the currently running rule istriggered, the control unit 170 (e.g., action execution module 179) iscapable of controlling execution of at least one action corresponding tothe condition at step 207.

For example, if at least one rule is executed, the control unit 170(e.g., the condition check module 179) is capable of monitoring todetect whether the condition specified in the rule for triggering theaction is fulfilled. If the condition or situation triggering the actionis fulfilled, the control unit 170 (e.g., the action execution module179) is capable of controlling the internal and/or external operationsfor executing the corresponding action. The action execution is capableof including the operations of executing the function (or applicationaccording to the predefined rule (e.g., condition and action),generating the execution result (e.g., context information), and feedingback the execution result to the user or others.

According to exemplary embodiments of the present invention, theoperation of defining the rule at step 201 may have been performedalready or may be performed by the user additionally before executingthe target rule. In the former case, the user is capable of inputtingthe instruction for executing the rule at step 203 immediately and, inthe latter case, the user is capable of performing steps 201 and 203 forboth defining the rue and inputting the instruction for executing therule.

FIGS. 3A to 3K are diagrams illustrating exemplary screens forexplaining an operation of generating a rule in a user device accordingto an exemplary embodiment of the present invention.

Referring to FIGS. 3A to 3K, exemplary operations of the control unit170 (e.g., the rule configuration module 173) for receiving the naturallanguage-based speech input made by the user and for defining and/oridentifying the rule (e.g., conditions and actions) according to thespeech input are illustrated.

FIG. 3A shows an exemplary screen of the user device 100 when the deviceuser executes the application for CAS according to an exemplaryembodiment of the present invention.

Referring to FIG. 3A, the CAS application provides a User Interface (UI)or Graphical User Interface (GUI) (hereinafter, referred to as a ‘screeninterface’) including a first menu 310 (e.g., my menu or ‘My Rule’ inFIG. 3A) for displaying a list of the rules defined by the user, asecond menu 320 (e.g., acting menu, ‘Acting Rule’ in FIG. 3A) fordisplaying a list of currently running rules among the defined rules,and a third menu 350 (e.g., add menu or ‘Add Rule’ in FIG. 3A) fordefining new rules additionally.

The screen interface is capable of providing a list corresponding to themenu item selected in the first menu 310 and the second menu 320. Asshown in FIG. 3A, if the first menu 310 is selected, a list of items(e.g., “home” item 330 and “Taxi” item 340) corresponding to the rulesdefined by the user.

The items 330 and 340 may be provided with a full-down menu item 335 fordisplaying the detailed information configured with the correspondingitem. For example, if the user selects the full-down (e.g., drop down)menu item 335, a full-full down window appears below the correspondingitem.

In the state of FIG. 3A, the user is capable of selecting (tapping) thethird menu (Add Rule) 350 for defining a new rule. Then the control unit170 of the user device 100 (e.g., the rule configuration module 173)determines whether to initiate the operation for defining the rule andswitches to the rule configuration mode along with displaying acorresponding screen interface. FIG. 3B shows an exemplary screeninterface displayed in such a case.

FIG. 3B shows an exemplary screen of the user device 100 when the deviceuser executes the rule configuration mode for defining the rule. In anexemplary embodiment of the present invention, the operation depicted inFIG. 3B is of providing the user with a tutorial on the method fordefining a rule, and this tutorial provision step may be omittedaccording to the user's intention.

Referring to FIG. 3B, the tutorial can be provided in the form of apopup window 351. The control unit 170 is capable of controlling todisplay the tutorial in the form of the popup window 351 presenting theguidance (e.g., picture and text) for guiding how to define a rule. Forexample, the guidance may be provided with an image 351 c indicatingactivation of speech recognition mode and a text 351 d guiding how todefine a rule (e.g., how about to make a rule like “play music if‘subway’ is spoken”, “do the following if ‘subway’ is spoken”).

The popup window 351 providing the tutorial may include a menu item 351a to confirm defining the rule (e.g., ‘start’ button) and a menu item351 b to cancel defining the rule (e.g., ‘cancel’ button). The user iscapable of continuing or cancelling the defining of the new rule byselecting one of the menu items 351 a and 351 b of the popup window 351providing the tutorial as shown in FIG. 3B.

FIGS. 3C to 3K show the operations of defining a new rule in the ruleconfiguration mode according to an exemplary embodiment of the presentinvention. FIGS. 3C to 3K show exemplary screens displayed in receivingthe user's natural language-based speech and configuring conditions andactions of the corresponding rule in response to the user's naturallanguage-based speech

As depicted in FIGS. 3C to 3K, the control unit 170 displays the popupwindow 353 prompting the user to make a speech input (e.g., “speak arule”) in the rule configuration mode and waits for user speech input.In the state of FIG. 3C, the user is capable of making a naturallanguage-based speech input for at least one condition and at least oneaction corresponding to the condition based on the type of rule to bedefined (e.g., singular or plural structure).

For example, the user may make a speech input of “do following if‘subway’ is spoken” in the state of FIG. 3D. Then the control unit 170recognizes the user's speech input and displays the popup window 355 asa result of the speech recognition. According to an exemplary embodimentof the present invention, the control unit 170 may display the popupwindow 355 prompting the user to make a speech input with a notificationmessage (e.g., “what can I do”) for action to be performed when thecondition “subway” is fulfilled with the speech recognition result“command [subway] what can I help you?” and then wait for user's speechinput.

In the state of FIG. 3D, the user can make a speech of “turn on Wi-Fi”.As illustrated in FIG. 3E, the control unit 170 recognizes the user'sspeech input and displays the popup window 357 notifying of speechrecognition mode and the progress (e.g., “recognition is in progress”)of the operation mapping the condition (e.g., subway) and action (e.g.,Wi-Fi turn-on). In various exemplary embodiments of the presentinvention, the recognition operation-related screen display may beomitted.

Once the recognition and mapping operations have completed, the controlunit 170 may provide the recognition and mapping result in the form of apopup window 359 as shown in FIG. 3F. For example, the control unit 170is capable of displaying a popup window 359 notifying of the informationon the newly defined rule and the action specified in association withthe rule. According to an exemplary embodiment of the present invention,the control unit 170 may notify that the new rule has been generatedwith the condition of the subway for configuring Wi-Fi turn-on when thecondition is fulfilled along with menu items prompting the user toprogress the operation (e.g., ‘confirm’ and ‘cancel’). In the state ofFIG. 3F, the user may select the ‘confirm’ menu item to apply theconfigured rule or the ‘cancel’ menu item to cancel the configured rule.

In the state of FIG. 3F, if the user selects the ‘confirm’ menu item (ormakes a speech input), then as illustrated in FIG. 3G, the control unit170 may display the popup window 361 prompting the user to make the nextspeech input (e.g., “speak the next command”) and wait for user's speechinput. The user may make a speech input of “change to vibration” in thestate of FIG. 3G. As illustrated in FIG. 3H, the control unit 170 maythen recognize the speech input and display the popup window 363notifying of speech recognition mode and the progress (e.g.,“recognition is in progress”) of the operation mapping the condition(e.g., subway) and additional action (e.g., configuration of vibration).If the recognition and mapping operations have completed, the controlunit 170 may provide the operation result in the form of a popup window365 as shown in FIG. 3I. For example, the control unit 170 is capable ofdisplaying the popup window 365 notifying of the rule being configuredin response to the user's speech input and the action corresponding tothe rule. According to an exemplary embodiment of the present invention,the control unit 170 is capable of notifying that a new rule has beengenerated with the condition of subway for switching to the vibrationmode when the condition is fulfilled along with the menu items (e.g.,‘confirm’ and ‘cancel’) by means of the popup window 365. In the stateof FIG. 3I, the user may select the ‘confirm’ menu item to apply theconfigured rule or the ‘cancel’ menu item to cancel the configured rule.

In the state of FIG. 3I, if the user selects the ‘confirm’ menu item,then, as illustrated in FIG. 3J, the control unit 170 may display apopup window 367 prompting the user to make a speech input (e.g., “speaknext command”) and waits for speech input. In the state of FIG. 3I, theuser may makes a speech input of “end (or stop)”. Then the control unit170 recognizes the speech input and provides the information on thecondition specified in the rule defined through steps 3B to 3J and atleast one action corresponding to the condition as shown in FIG. 3K.

For example, the control unit 170 may display the condition ‘subway’ ofthe rule defined through the above described operations along with theactions ‘Wi-Fi turn-on configuration’ and ‘vibration mode switchingconfiguration’ mapped to the condition as shown in FIG. 3K. In this way,the newly defined rule can be added to the rule list as shown in FIG.3A. For example, the newly added rule may be displayed as ‘subway’ item360 in the list along with the previously defined items (e.g., ‘home’item 330 and ‘taxi’ item 340, and the ‘subway’ item 360 may be providedwith detailed information (e.g., conditions and actions). The screeninterface may also display various settings of the device. For example,the screen interface may display a Wi-Fi setting 371, a sound setting373, and the like. The settings of the device may be associated with anitem (e.g., item 330, item 340, and/or item 360).

As described above with reference to FIGS. 3A to 3K, at least one actioncan be mapped to one condition according to various exemplaryembodiments of the present invention. In an exemplary embodiment of thepresent invention, the CAS method is capable of supporting singular ruledefinition operation and plural rules definition operation. This can besummarized as follows.

The singular structure rule definition operation can be summarized asshown in Table 1.

TABLE 1 Rule: If condition (instruction), then action. Example: “Home”is spoken, turn on Wi-Fi. Mail is received, send text message and thelike.

The plural structure rule definition operation can be summarized asshown in Table 2.

TABLE 2 Rule: If condition 1(instruction1), then do following.   - Thenaction1.   - If condition2(command2), then action2.   - ...   - IfconditionN(instructionN), then actionN.   - end(stop). Example: “home”is spoken, do following.   - Turn on Wi-Fi.   - If inbound call isreceived, mute TV sound,   - If inbound call is received, blink LEDlamp,   - Switch to sound mode,   - stop.

As shown in Tables 1 and 2, a simple if statement such as <“home” isspoken, then switch to ring.> or a complex if statement such as <“home”is spoken, then mute TV sound upon receiving inbound call> can be used.According to an exemplary embodiment of the present invention, pluralactions (e.g., terminal function, situation-adaptive multi-appcessory(App+Accessory) interoperation, and use of the cloud service)corresponding to at least one condition based on the simple or complexif statement can be configured. Among the plural actions, the terminalfunction may include Wi-Fi mode configuration, ring/vibration/mute modeswitch, text messaging (recipient and content speech configuration),camera flash blinking, and the like; and the use of the cloud servicemay include checking (e.g., determining) user location (using GPS) andthen sending a text message, and the like.

The types of the conditions (or instructions) that may be specified inthe rule and the types of the actions that may be configured percondition can be summarized as shown in Table 3.

TABLE 3 Type of condition Type of action • Location: home, office, andthe • internal function of device  like.   -  call (answer in absence,place a • Event condition    call), send text, send mail   -  inside ofdevice: if   -  ring/vibration setting, flash    call/text/mail isreceived,    (camera), music playback, auto-    if 00 o'clock arrives,   screen rotation    and the like.   -  WiFi setting, Bluetoothsetting,   -  outside of device: if it    GPS, data network, AllShareCast    rains, and the like.   -  Sleep mode, block mode,   -  activity:going to work,    synchronization    leaving the office, driving, • inAppcessory interoperation    exercise, sleeping   -  TV control(power-off, volume • abstract condition (command)    control (mute,etc.) and the like)   -  if “I love you” is spoken,   -  Lamp control(on/off, blinking,   -  if “abracadabra” is    and the like)    spoken,• Cloud service   -  if “be quite” is   -  GPS service, Meteorological   spoken,    Administration (weather) service

According to various exemplary embodiments of the present invention, theuser device 100 is capable of performing interaction (e.g., query andanswer) by providing voice or text feedback to the user about theinformation necessary depending on the action the user intends toexecute when a specific condition is fulfilled. According to anexemplary embodiment of the present invention, the information to bereceived from the user for all the actions supportable by the userdevice 100 may be provided in the form of a DataBase (DB). According toan exemplary embodiment of the present invention, in the case of thetext message transmission function, the user device 100 may recognizethe necessity of additional information about the recipient and textmessage and may prompt the user to input the additional information inthe form of speech or text, alarms erroneous input, and asks forreceiving re-input. Such an operation is described hereinafterexemplarily with reference to FIGS. 4A to 4J.

FIGS. 4A to 4J are diagrams illustrating exemplary screens forexplaining an operation of generating a rule in a user device accordingto an exemplary embodiment of the present invention.

Referring to FIGS. 4A to 4J, the control unit 170 is capable ofdisplaying a popup window 451 prompting the user to make a speech inputin response to the rule configuration mode entry in the state of FIG. 3A(or FIG. 3B) and waiting for speech input. In the state of FIG. 4A, theuser may make a natural language-based speech input for configuring atleast one condition and at least one action per condition according tothe type (e.g., singular structure and plural structure) of the rule tobe defined.

For example, in the state of FIG. 4A, the user may make a speech inputof “if taxi is spoken then do the following.” Then the control unit 170may recognize the user's speech input and provide the recognition resultin the form of a popup window 453 as shown in FIG. 4B. According to anexemplary embodiment of the present invention, the control unit 170 maydisplay the popup window 453 prompting the user to make a speech inputfor an action to be performed with the fulfillment of the condition“taxi” along with the speech recognition result such as “command [taxi]what can I do?” and wait for speech input.

In the state of FIG. 4B, the user may make a speech input of “sendmessage.” Then the control unit 170 may recognize the speech input andcheck the information required to be received from the user for theaction (e.g., “send message”) so as to perform feedback as shown in FIG.4F.

For example, as shown in FIG. 4C, the control unit 170 may display apopup window 455 prompting the user to input “recipient” as theinformation required for the action of “send message” (e.g., “command[taxi] speak recipient”) and waits for speech input of the user. In thestate of FIG. 4C, the user may make a speech input of a specificrecipient (e.g., “boyfriend”) to which the message is sent. The controlunit 170 may recognize the speech input by the user, prompt the user toinput “message content” as the information for the action “send message”as shown in FIG. 4D (e.g., “To. [boyfriend] speak content”) in the formof a popup window 457, and wait for speech input of the user.

In the state of FIG. 4D, the user may input the message content to besent to the recipient (e.g. “boyfriend”). According to an exemplaryembodiment of the present invention, the user may make a speech input of“send my location” as the message content to be sent to the “boyfriend.”Then the control unit 170 may recognize the user's speech input, checkthe necessity of additional information for executing the action of“send location”, display a popup window 459 prompting the user to enterthe time interval for executing the action of “send location” (e.g. “To.[boyfriend] send how often?”), and wait for receiving speech input.

In the state of FIG. 4E, the user may input a transmission interval forexecuting the action of “send my location.” According to an exemplaryembodiment of the present invention, the user may make a speech input of“every 5 minutes” as transmission interval for executing the action of“send my location.” Then the control unit 170 may recognize the speechinput and display the recognition result as shown in FIG. 4F.

For example, as shown in FIG. 4F, the condition information 410configured for the rule to be defined by the user and the actioninformation 411, 413, 415, and 417 corresponding to the condition may beprovided in the form of the popup window 461. Particularly in the caseof the action information 411, 413, 415, and 417, auxiliary actionsconfigured according to the initially configured action may be provided.According to an exemplary embodiment of the present invention, the firstaction may be set to “send message” as denoted by reference number 411,the second action as the target to which the message is transmittedaccording to the first action to “boyfriend” as denoted by referencenumber 413, the third action as the content of the message to betransmitted to the boyfriend according to the first action and/or secondaction to “send my location” as denoted by reference number 415, and thefourth action as the transmission interval of the message including mylocation to be transmitted the boyfriend according to the first action,second action, and/or third action to “5 minutes” as denoted byreference number 417. For example, according to an exemplary embodimentof the present invention, the control unit 170 may ask the user for theinformation necessary to execute the previous action in voice or textinteraction. At this time, the control unit 170 may recognize the actionrequiring additional information no longer and may provide theinformation on the requested actions (e.g., “send message” 411,“boyfriend” 413, “send my location” 415, and “5 minutes” 417 configuredfor the condition (e.g. “taxi” 410)) in the form of the popup window 461as shown in FIG. 4F. In this way, the user is capable of defining therule for action of transmitting a message including my location to theboyfriend at every 5 minutes when the condition “taxi” is fulfilled.

Meanwhile, in the state of FIG. 4F, the user may select the “confirm”menu item to apply the definition on the actions (e.g., boyfriend,sending my location, and 5 minutes) corresponding to the condition (e.g.taxi) of the rule configured through the above procedure or the “cancel”menu item for cancel or reconfigure the rule.

In the state of FIG. 4F, if the user selects a confirm menus item (ormake a speech input by speaking “confirm”, then, as illustrated in FIG.4G, the control unit 170 may display a popup window 463 prompting theuser to make a speech input (e.g., “please speak next command”) and waitfor speech input. In the state of FIG. 4G, the user may make a speechinput of “switch to vibration” for switching to the vibration mode asadditional action corresponding to the condition (e.g., taxi). Then thecontrol unit 170 may recognize the user's speech input and display thespeech recognition mode indicator, the condition (e.g., taxi) inputpreviously, and the additional action prompt in the form of a popupwindow 465 as shown in FIG. 4H. According to an exemplary embodiment ofthe present invention, as shown in the popup window 465, the controlunit 170 may notify of the generation of a new rule with the condition“taxi” for switching to the vibration mode when the condition isfulfilled along with menu items (e.g., “confirm” and “cancel”).

In the state of FIG. 4H, if the user selects the ‘confirm’ menu item,then as illustrated in FIG. 4I, the control unit 170 may display a popupwindow 467 prompting the user to make the next speech input (e.g.,“please speak next command”) and wait for user's speech input. The usermay make a speech input of “end (or “stop”) to end additionconfiguration of the rule in the state of FIG. 4I. Then the control unit170 may recognize the user's speech input and provide the information onthe condition specified in the rule configured through steps of FIGS. 4Ato 4I and at least one action corresponding to the condition as shown inFIG. 4J.

For example, as shown in FIG. 4J, the control unit 170 may provide ascreen notifying that the rule has the condition “taxi” for the actions“send message” and “sound setting” along with the detailed information“send message including my location to boyfriend at every 5 minutes” and“sound is set to vibration.”

Meanwhile, according to various exemplary embodiments of the presentinvention, as shown in FIGS. 4A to 4F, the control unit 170 may ask theuser for the information required additionally in voice or textdepending on the action input by the user. According to variousexemplary embodiments of the present invention, as shown in FIGS. 4G to4I, the control unit 170 may recognize the action requiring noadditional action (e.g., sound setting) and skip requesting the user foradditional information to jump to the next step.

Hereinabove, the descriptions have been made of the operations ofdefining the rule according to the various exemplary embodiments of thepresent invention. A description has been made of the exemplaryoperation of executing the above-defined rule hereinafter. According tovarious exemplary embodiments of the present invention, the predefinedrules may be executed immediately in response to the user's speech ortext input as described above. Also, the predefined rule may generate awidget according to the user's definition in the user devices andexecute the corresponding rule by means of the widget as to be describedherein below. For example, according to various exemplary embodiments ofthe present invention, the instruction for the rule may be executed bymeans of the widget.

In various exemplary embodiments of the present invention, if a certainoperation such as receiving an inbound call interrupts generating arule, the rule generation operation progressed until then is saved (ortemporarily stored) to process the operation caused the interrupt.

FIGS. 5A to 5E are diagrams illustrating exemplary screens forexplaining an operation of executing a predefined rule in a user deviceaccording to an exemplary embodiment of the present invention.

FIGS. 5A to 5E show exemplary operations of receiving, at the ruleexecution module 175 of the control unit 170, the natural language-basedspeech input made by the user and executed the rule in response to thespeech input, checking, at the condition check module 177 of the controlunit 170, the condition specified in the rule, and executing, at theaction execution module 179 of the control unit 170, at least one actioncorresponding to the condition when the condition is fulfilled (e.g.,when the context is detected).

Referring to FIGS. 5A to 5E, FIG. 5A shows an exemplary screen of theuser device 100 when a widget for CAS is provided according to anexemplary embodiment of the present invention.

As shown in FIG. 5A, the CAS widget 500 may be displayed on the homescreen (or menu screen) of the user device 100. The widget 500 may beprovided with an instruction input region (or rule execution button) 510for the user to make an input (e.g., tap and touch) for executing therule and an execution information region showing information on thecurrently running rules among the user-defined rules. The widget 500also may be provided with a refresh function item 530 for updating theinformation on the currently running rules. According to an exemplaryembodiment of the present invention, the instruction input region 510may be provided with an image or text in consideration of intuitivenessfor the user. FIG. 5A is directed to an exemplary case in which theexecution information region 520 indicates that no rule is currentlyrunning.

In the state of FIG. 5A, the user may select (e.g., touch gesture ortap) the instruction input region 510 for executing the rule. Asillustrated in FIG. 5B, the control unit 170 (e.g., the rule executionmodule 175) may then execute the rule with the display of the popupwindow 551 prompting the user to input (or speak) the information on therule to be executed (e.g. “speak command”) and wait for user's speechinput.

In the state of FIG. 5B, the user may make a speech input (e.g.,“subway”) for the rule to be executed. As illustrated in FIG. 5C, thecontrol unit 170 may then recognize the user's speech input and providea popup window 553 notifying that the rule corresponding to “subway” isbeing loaded (or recognized). In various exemplary embodiments of thepresent invention, the recognition progress screen display may beomitted.

If the recognition and loading operation has completed, the control unit170 may provide a recognition and a loading result in the form of apopup window 555. For example, the control unit 170 may provide theinformation on the rule to be executed according to the user's speechinput and condition specified in the rules and actions corresponding tothe condition by means of the popup window 555. According to anexemplary embodiment of the present invention, the control unit 170 mayprovide notification that the rule to be executed is “subway” configuredwith the condition of “subway” (e.g. execute [subway]) having theactions of “Wi-Fi turn-on” and “switch to vibration” (e.g. ‘Wi-Fi’ ison, ‘vibration’ is configured). In various exemplary embodiments of thepresent invention, the rule information screen display may be skippedand the procedure may jump to the operation to the operationcorresponding to FIG. 5E. The screen of FIG. 5E may be displayed for apredetermined duration and then followed by the corresponding operation.

When a predetermined time duration elapses in the state of FIG. 5D, thecontrol unit 170 may provide the information (e.g. subway) on the ruleexecuted by the user (or current running rule) in the executioninformation region 520 of the widget in the form an image or text. Forexample, the execution information region 520 of the widget 500 maypresent an icon or text indicating “subway” instead of the message of“there is no rule running currently” as shown in FIG. 5E. Also, thecontrol unit 170 may provide a notification item 550 to notify of thecurrently running rules in the indicator area providing variousoperation state of the user device 100 in response to the execution ofthe rule as shown in FIG. 5E. A description is made of the notificationitem later.

Although FIG. 5E is directed to the exemplary case in which only onerule is running, a plurality of rules may be running and, in this case,the execution information region 520 may provide the information on theplurality of currently running rules. Although the description isdirected to an exemplary case of prompting the user to make a speechinput in selection of the instruction input region 510, it may bepossible to provide a list of the rules defined by the user in selectionof the instruction input region 510 in order for the user to select atlease on rule.

The control unit 170 (e.g., the condition check module 177) also mayoperate to determine whether the condition specified in the currentlyrunning rule, among the various rules defined through the aboveprocedure, is fulfilled. The control unit 170 (e.g., the actionexecution module 179) may operate to execute at least one action mappedto the condition of the rule which is fulfilled (e.g., at least oneaction mapped to the context corresponding to the condition of therule).

As described above, according to the various exemplary embodiment of thepresent invention, the rule may be executed by means of the widget 500.According to an exemplary embodiment of the present invention, the usermay select the instruction input region (or rule execution button) 510and speak the configure rule (or command). Then the control unit 170 mayperform popup text or voice feedback to the user to provide theinformation on the rule start time and actions to be executed. Thecontrol unit 170 also may add the corresponding rule to the executioninformation region 520 and display the notification item 550 indicatingthat there is a currently running rule in the indicator region. Adescription is made of the notification item 550 later.

Hereinafter, the detailed operations of the CAS control methodsaccording to various exemplary embodiments of the present invention aredescribed with reference to accompanying drawings (e.g., FIGS. 6 to 13).

FIG. 6 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention.

FIG. 6 shows an exemplary situation in which the user configures andexecutes a rule through speech interaction, and the user device 100detects fulfillment of the condition specified in the rule and executesthe action triggered when the condition is fulfilled. Particularly inFIG. 6, the user executes action1 (e.g., sound setting) corresponding tocondition1 (e.g., home) and performs action2 (e.g., blink lamp) andaction3 (e.g., mute TV sound) under a sub-condition (condition2) (e.g.,when receiving call).

Referring to FIG. 6, the user may define a rule using the user device100. For example, the user may activate the function of generating arule and define the rule for the user device 100 to change the ringtoneat a specific location and blink the lamp and mute TV sound upondetection of incoming call at the specific location, by manipulating theuser device 100. According to an exemplary embodiment of the presentinvention, the user may define the rule of “switching the indicationmode to the ring mode at home and blinking the lamp and mute TV sound inreceiving inbound call at home” in the stepwise interaction with theuser device 100. The operation of defining the rule may be performedthrough the rule generation procedure described with reference to FIGS.3A to 3K and 4A to 4K. The operation of defining the rule may beprogressed through natural language-based interaction.

The user may instruct execution of the rule defined through naturallanguage-based speech or text interaction. The situation to be detected(e.g., condition) may be “[home] is spoken or a call is received fromhome”, and the action to be taken in fulfillment of the condition may be“setting sound blinking, mute TV sound”. Although not defined separatelyin the case of FIG. 6, the user device 100 may control execution ofextra action depending on the executed action. For example, insubsequence to blinking lamp and muting TV sound, another action of“recovering the previous state of the lamp upon receipt of the call andplaying TV sound upon end of the call session” may be performed.

In the state in which a rule has been defined, the user may execute, ifnecessary, the defined rule. For example, the user may make a speechinput by speaking “home” through the process as described in FIGS. 5A to5E when entering the home from outdoor. Then the user device 100 maycheck (e.g., determine) whether a there is a rule corresponding to“home” among the currently running rules. If the rule corresponding to“home” is running, the device 100 may check (e.g., determine) at leastone condition specified in the rule “home” and actions corresponding tothe condition.

The user device 100 recognizes “home” as the first condition defined inthe rule “home” and “ring mode switching” as the first actioncorresponding thereto. Accordingly, the user device 100 switches theindication mode of the user device 100 to the ring mode in reaction tothe user's speech input of “home” as denoted by reference number 610.

The user device 100 also may recognize the second condition “whenreceiving a call” specified in the rule “home” and check the interruptto the condition (e.g., receipt of inbound call). Afterward, if aninbound call is received as denoted by reference number 620, the userdevice 100 recognizes the second condition “when receiving a call”specified in the rule “home” in reaction to the interrupt. The userdevice 100 may operate to control the second action “blinking lamp” asdenoted by reference number 630 and the third action “muting TV sound”as denoted by reference number 640.

If the user accepts the receipt of the inbound call (e.g., call sessionis established) in the state the ringtone (e.g., a phone bell sound)indicating receipt of voice call is played, the user device 100 mayrecover the lamp to the previous state and, if the call session isreleased, releases the mute of the TV sound.

FIG. 7 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention.

FIG. 7 shows an exemplary situation in which a user configures andexecuted a rule through speech interaction, and the user device 100recognizes the condition specified in the rule and executes the actiontriggered when the condition is fulfilled. Particularly in FIG. 7, theuser may define a rule of transmitting the user location (or location ofthe user device 100) to at least one target user device automatically ata predetermined interval. The user device 100 may execute the rule inresponse to a user input and perform the action of transmitting thelocation information to the at least one target user device at aninterval specified in the execution rule.

Referring to FIG. 7, the user may define the rule using the user device100. For example, the user may activate the rule generation function (orapplication) by manipulating the user device 100 and define a rule oftransmitting the location information to at least one target user device200 at a predetermined interval. According to an exemplary embodiment ofthe present invention, the user may generate a rule of “if I take ataxi, then send my location information to my father and little brother(or sister) at every 5 minutes.” At this time, the rule may be generatedthrough speech interaction with the microphone 143 or text interactionwith the input unit 120 or touchscreen 130 as to be described later. Itis preferred that the speech and text interaction is based on a naturallanguage as described above. The situation to be detected (e.g.,condition), specified in the defined rule may be “when movement occurs”,and the action to be taken in fulfillment of the condition may be “sendlocation information to father and little brother (sister) at every 5minutes.”

The user device 100 may provide an interface for designating at leastone target user device 200 for transmitting the location information andmapping the information on at least one target device 200 (e.g., phonenumber, name, and nickname) to the rule. The rule may be predefined andredefined by the user if necessary anytime anywhere.

The user may input an instruction for executing the rule in the form ofspeech, text, or gesture through an interface given in defining therule. Then the user device 100 may map the instruction to the definedrule and store the mapping information. In the case of using the speechinput, the user device 100 may store the waveform of the speechinstruction, convert the speech to a text and store the text, or storeboth the waveform of the speech and converted text.

In the state in which the rule has been defined, the user may identify,activate, and/or execute the rule with the predefined instruction (e.g.speech, text, and gesture), if necessary. For example, the user mayinput the corresponding instruction as a speech input (e.g., “taxi”,“taxi mode”, and “take taxi now”) right before or upon getting in a taxias in the exemplary case of FIGS. 5A to 5E. Although the description isdirected to the case in which the instruction is the speech instructioninput through the microphone 143, the instruction may be input in theform of text by means of the input unit 120 or touchscreen 130 or in theform of gesture. The speech and text instructions may be the naturallanguage-based instructions as described above.

When the user intends to make a speech input for executing the rule (orcondition or situation recognition), the user may take a preliminaryaction for notifying the user device 100 of the use of speech input forexecuting the rule in advance. In an exemplary case of executing therule (or condition or situation recognition) through speech, it may benecessary to activate the microphone 143. This is because if themicrophone 143 is always on an unintended sound input may causeunnecessary operation or error. Accordingly, it is preferred to define aspecific action (e.g. widget, gesture, and function key manipulations)for activating the microphone 143 in the speech input mode such that theuser to take that action to turn on the microphone 143 before speechinput. According to an exemplary embodiment of the present invention,the user may speak the instruction after a predetermined gesture,pushing a predetermined function key, or selecting a rule executionbutton of a widget.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. For example, the user device 100may search the predefined rules (e.g., speech waveforms mapped to therespected rules) for the speech waveform corresponding to the speechinput. The user device 100 also may convert the input speech to a textto retrieve the text among the predetermined rules (e.g., texts mappedto the respective rules). The user device 100 may search for both thespeech waveform and text among the predetermined rules (e.g., waveformsand texts mapped to the respective rules).

The user device 100 may perform condition (situation) recognitionaccording to the rule execution. For example, the user device 100 maydetect the fulfillment of condition1 such as “take taxi now” based onthe predefined rule and check condition2 such as predetermined time(e.g., 5 minutes) upon fulfillment of condition1. In this case, the userdevice 100 may operate to perform action1 for checking location of theuser device 100 at every time specified in condition2. The user device100 also may operate to perform action2 for sending the locationinformation to at least one target user device 200 according to action1.

Meanwhile, the at least one target user device 200 may perform feedbackof the location information on the user device 100 through apredetermined interface upon receipt of the location information fromthe user device 100. For example, the target user device 200 may displaythe location information on the user device 100 on a map image as shownin FIG. 7. At this time, the map image and location information may betransmitted as processed by the user device 100, or the locationinformation transmitted by the user device 100 is presented on a mapimage provided by the target user device 200.

As described above, the user may notify at least one other designateduser of the user's location at a predetermined time interval accordingto the exemplary embodiment of FIG. 7. The at least one other user mayacquire the location and moving path of the user without extra action.

FIG. 8 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention.

FIG. 8 shows an exemplary situation in which the user configures andexecutes a rule using a composite application, and the user device 100recognizes the condition specified in the rule and executes an actiontrigged when the condition is fulfilled. Particularly in FIG. 8, theuser may define a rule of transmitting the user location (or location ofthe user device 100) to at least one target user device in response toan external event. The user device 100 executes the rule according tothe user input and executes the action of transmitting the locationinformation to the target user device 200 upon receipt (e.g., oroccurrence) of the event from the target user device 200 specified inthe execution rule. FIG. 8 shows an exemplary case in which the userdevice 100 detects an input (event) from the external device (e.g.,target user device 200) and executes the predetermined action.

In various exemplary embodiments of the present invention, the compositeapplication may be the application of modularizing the screen forproviding the end user with diverse information received from variousresources in most preferred way (e.g., as required by user), anddesigning the screen configuration and screen mode switching dependingon the user's right and role so as to optimize the user's experience.

Referring to FIG. 8, the user may define the rule using the user device100. For example, the user may activate the rule generation function (orapplication) and define a rule of transmitting the location informationto at least one target user device 200 whenever an event (e.g., receiptof message) occurs. According to an exemplary embodiment of the presentinvention, the user may generate a rule of “if a call is received at thewheel from wife, send my current location information.” At this time,the rule may be generated through speech interaction with the microphone143 or text interaction with the input unit 120 or touchscreen 130 as tobe described later. It is preferred that the voice and text interactionis based on a natural language as described above. The situation to bedetected (e.g., condition), specified in the rule specified as shown inFIG. 8 may be “if a call is received at the wheel from wife,” and theaction to be taken in fulfillment of the condition may be “send mycurrent location information.” Also, an additional condition such as “ifa message including a phrase asking for location such as ‘where’ at thewheel from wife” can be further configured.

The user device 100 may provide an interface for designating at leastone target user device 200 generating event and mapping the informationon at least one target device 200 (e.g., phone number, name, andnickname) to the rule. The rule may be predefined and input or redefinedby the user if necessary anytime anywhere, and the instruction forexecuting the rule may be input through a given interface in the form ofspeech, text, or gesture. Then the user device 100 may map theinstruction to the defined rule and store the mapping information.

In the state in which the rule has been defined, the user may executethe rule with the predefined instruction (e.g., speech, text, andgesture), whenever necessary. For example, the user may input thecorresponding instruction as a speech input (e.g., “drive”, “drivemode”, and “I will drive”) right before or upon getting in a car as inthe exemplary case of FIGS. 5A to 5E. Although the description isdirected to the case in which the instruction is the speech instructioninput through the microphone 143, the instruction may be input in theform of text by means of the input unit 120 or touchscreen 130 or in theform of gesture. It is preferred that the speech and text instructionsare the natural language-based instructions as described above. When itis intended to make a speech input, the user may take a preliminaryaction (e.g., turning on the microphone 143) for notifying the userdevice 100 of the use of speech input for executing the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 may alsodetect the fulfillment of the condition (situation). For example, theuser device 100 may detect the fulfillment of condition1 such as “I willdrive” specified in the defined rule and check condition3 such asreceipt of text message (text message including specific condition suchas “where”) from condition2 such as designated target device 200according to conditon1. For example, if a text message fulfillingcondition3 is received according to the fulfillment of condition2, e.g.,from the target user device 200, the user device 100 may perform action1for acquiring the location information on the user device 100. The userdevice 100 also may perform action2 of transmitting the locationinformation acquired according to action1 to the target user device 200.

If the location information transmitted by the user device 100 isreceived, the target user device 200 may perform feedback of thelocation information on the user device through a predeterminedinterface. For example, the target user device 200 may display the userdevice location on a map image as shown in FIG. 8.

As described above, according to the exemplary embodiment of FIG. 8, ifan event is received from a designated target user, the user may notifythe target user of the user location through a text message withouttaking extra action.

FIG. 9 is a diagram illustrating an exemplary situation of providing CASusing a user device according to an exemplary embodiment of the presentinvention.

FIG. 9 shows an exemplary situation in which one of the similar rules isselected by the user or recommended programmatically or executingsimilar rules simultaneously. Particularly in the exemplary situation ofFIG. 9, the user may define a rule for the user device 100 to performfeedback of alarm or control a specific function. The user device 100may execute the rule in response to the user input and control aspecific function of the user device 100 or execute an alarming actionupon detection of change in the situation as specified in the rule.

Referring to FIG. 9, the user may define a rule using the user device100. For example, the user may generate a rule of outputting an alarmupon detection of an event caused by change in environment (hereinafter,first rule) and a rule of controlling a function of the user device 100upon detection of an event caused by change in environment (hereinafter,second rule) with the activation of the function (or application)capable of generating the rules by manipulating the user device 100.According to an exemplary embodiment, it is possible to generate thefirst rule such as “output alarm when driving speed is equal to orgreater than 80 Km/h” and the second rule such as “increase audio volumeof the car or the user device 100 when the driving speed is equal to orgreater than 60 Km/h.” At this time, as described above, the rule may bedefined by speech input through the microphone 143 or text input throughthe input unit 120 or touchscreen 130. The speech and text input may bemade with a natural language. The situation to be detected (e.g.,condition), may be “when change in environment (e.g., driving speed) isequal to or greater than a predetermined threshold,” and the action tobe taken in fulfillment of the condition may be “output alarm or controlaudio volume.”

As described above, the first and second rules may be predefined orgenerated and redefined by the user in real time, whenever necessary,anytime anywhere; and the instruction for executing the rule may beinput in the form of speech, text, or gesture through a given interface.Then the user device 100 may map the defined rule and the instructionand store the mapping as described above. FIG. 9 shows an exemplary casein which plural rules such as first and second rules are mapped to thesame instruction.

In the state in which the rules (the first and second rules) have beendefined, the user may execute the rules using the defined instructions(e.g., speech, text, and gesture) whenever necessary. For example, theuser may make an instruction (e.g., speech input such as “drive”, “drivemode”, and “I will drive”) before or upon getting in a car as in theexemplary case of FIGS. 5A to 5E. Although the description is directedto the case in which the instruction is the speech instruction inputthrough the microphone 143, the instruction may be input in the form oftext by means of the input unit 120 or touchscreen 130 or in the form ofgesture. As described above, when the user intends to make a speechinput, the user may take a preliminary action (e.g., turning on themicrophone 143) for notifying the user device 100 of the use of speechinput for executing the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 also maydetect the fulfillment of the condition (situation). For example, theuser device 100 detects the fulfillment of condition 1 such as “I willdrive” specified in the defined rule and then check whether theconditions are fulfilled (e.g., whether driving speed is equal to orgreater than 60 Km/n or 80 Km/n). If the second condition is fulfilled,the user device 100 may take action1. For example, if the driving speedis equal to or greater than 60 Km/h, the user device 100 may take anaction of increasing the audio volume thereof or the car). Also, if thedriving speed is equal to or greater than 80 Km/h, the user device 100may take an action of outputting the alarm sound according to the firstrule.

Meanwhile, in the case in which there are multiple conditions (e.g., thefirst and second rules) matching the instruction, the user device 100may recommend one of the conditions for user's selection. As shown inthe exemplary situation of FIG. 9, the user device 100 may display apopup window presenting the first condition (≥80 Km/h) and the secondcondition (≥60 Km/h) to prompt user selection.

The previous exemplary embodiment has been directed to an exemplarysituation of executing the rule before start of driving. Accordingly,when the predetermined rule with multiple conditions is running, theuser device 100 may monitor the driving speed to determine whether thefirst and second conditions are fulfilled and execute the actionscorresponding to the conditions to both the fulfilled conditionssequentially or the action corresponding to the lastly fulfilledcondition.

Meanwhile, the user may execute the predefined rule at the wheel (e.g.,in the state of driving at 110 Km/h as shown in FIG. 9).

In this case, the user device 100 may recognize and parse the speechinput to execute the rule indicated by the speech input. The user device100 may also detect the fulfillment of the condition (situation). Forexample, the user device 100 detects the fulfillment of condition 1 suchas “I will drive” specified in the defined rule and then check (e.g.,determine) whether the second condition is fulfilled (e.g., whetherdriving speed is equal to or greater than 60 Km/n or 80 Km/n). Becausethe current driving speed of 110 Km/h fulfills both the first and secondconditions, the user device 100 may execute the actions corresponding tothe first and second conditions simultaneously. According to anexemplary embodiment of the present invention, when the first and secondconditions are fulfilled (e.g., the current driving speed of 110 Km/h isequal to or faster than 60 Km/h and 80 Km/h), the user device 100 mayincrease the audio volume thereof or the car and output alarm soundsimultaneously.

In the case that the instruction matches a rule specifying multipleconditions, the user device 100 may recommend one of the conditions foruser's selection. As shown in the exemplary situation of FIG. 9, theuser device 100 may display a popup window presenting the firstcondition (80 Km/h) and the second condition (60 Km/h) to prompt userselection when the current situation (100 Km/) fulfills both the firstand second conditions. By taking notice that the user is at the wheel,it is preferred that the condition selection is made through speechinput. According to the user setting, the condition selection may beperformed in the form of text input and gesture input as well as speechinput.

FIG. 10 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention.

FIG. 10 shows an exemplary situation of providing CAS using the userdevice 100 and an external device (or an object capable of communicationwith the user device 100 or to which a device capable of communicatingthe user device 100 is attached). In the exemplary case of FIG. 10, arule may be defined to check (e.g., determine) change of the externalenvironment at a predetermined interval set by the user and feeds backan alarm depending on the check result. The user device 100 may executethe predefined rule in response to the user instruction and take anaction of alarming when the environmental change specified in the ruleis detected.

Referring to FIG. 10, the user may define the rule using the user device100. For example, the user may activate the function (or application)capable of generating a rule by manipulating the user device 100 anddefine the rule for checking the event occurring depending on theexternal environmental change at a predetermined interval and output analarm triggered by the event. According to an exemplary embodiment ofthe present invention, the user may generate a rule such as “show remindmessage and turn on the bathroom light if not take medicine at every 4hours except between 10:00 PM and 7:00 AM.” The rule may be generated byspeech input through the microphone 143 or text input through the inputunit 120 or touchscreen 130. The speech and text input may be made witha natural language. In the case of FIG. 10, the situation to be detected(e.g., condition), may be “if not take medicine at every 4 hours” and“except between 10:00 PM and 7:00 AM” and the action to be taken infulfillment of the condition may be “show remind message and turn on thebathroom light.”

As described above, the rule may be predefined or generated andredefined by the user in real time, whenever necessary, anytimeanywhere; and the instruction for executing the rule may be input in theform of speech, text, or gesture through a given interface. Then theuser device 100 may map the defined rule and the instruction and storethe mapping as described above.

In the state in which the rule has been defined, the user may executethe rule using the defined instruction (e.g., speech, text, and gesture)whenever necessary. For example, the user may make an instruction (e.g.,speech input such as “medicine”, “check medicine bottle”, and“medication time”) to execute the predefined rule as in the exemplarycase of FIGS. 5A to 5E. Although the description is directed to the casein which the instruction is the speech instruction input through themicrophone 143, the instruction may be input in the form of text bymeans of the input unit 120 or touchscreen 130 or in the form ofgesture. As described above, when the user intends to make a speechinput, the user may take a preliminary action (e.g., turning on themicrophone 143) for notifying the user device 100 of the use of speechinput for executing the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 also maymonitor to detect the fulfillment of a condition (situation) specifiedin the executed rule. For example, the user device 100 may monitor todetect the context corresponding to the activated rule. As anotherexample, the user device 100 may detect the fulfillment of condition1such as “medication time” specified in the defined rule and then checkthe condition2 (e.g., 4 hours) which is followed by condition3 such aswhether an external device (e.g., medicine bottle) is moved (shaken). Invarious exemplary embodiments of the present invention, the user device100 may check (e.g., determine) the movement of the external devicethrough radio communication. In order to accomplish this, the externaldevice (e.g., medicine bottle) may have a communication module (e.g.,Bluetooth Lower Energy (BLE) tag, RF tag, NFC tag, and the like) capableof communication with the user device 100.

If no movement of the external device (e.g., medicine bottle) isdetected for predetermined time duration (e.g., 4 hours), the userdevice 100 may operate to execute action1 such as outputting a remindmessage of “take medicine!” The user device 100 may execute action2 ofcontrolling a target device (e.g., lamp, refrigerator, electric pot, andthe like) as well as the output of the remind message as action1.

In an exemplary embodiment of the present invention, the target devicemay be any of objects used in everyday life such as lamp, refrigerator,and electric pot, as well as smart devices, requiring any action at aspecific situation. If the target device is of being capable ofcommunicating with the user device 100 as in FIG. 10, controlling thetarget device directly and, otherwise, indirectly via an auxiliarymodule (e.g., communication-enabled power control device) may bepossible. According to an exemplary embodiment of the present invention,the user device 100 may communicate with the power control device havingcommunication function to supply power to the bathroom lamp as a targetdevice.

FIG. 11 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention.

FIG. 11 shows an exemplary case in which an owner of a user deviceinputs an instruction for executing a rule, and in which a child usermay not execute the rule as a result of a lack of dexterity according toan exemplary embodiment of the present invention.

FIG. 11 shows an exemplary situation of providing CAS in which the ruleis configured and executed using a composite application, and the userdevice 100 detects the fulfillment of a condition specified in the ruleand executes the action corresponding to the condition. Particularly inthe exemplary case of FIG. 11, the user may define the rule of sending aspecific target user device 200 the location (e.g., location of the userdevice) in detection of an external event. The user device 100 mayexecute the rule according to a user input and, when an event isreceived from the target user device 200 specified in the executed rule,executes the action of transmitting photo information and locationinformation. In FIG. 11, the user device 100 detects the input (event)from the external device (e.g., target user device 200) and executes adesignated action.

Referring to FIG. 11, the user may define a rule using the user device100. For example, the user (or parent of the user) may activate the rulegeneration function (or application) by manipulating the user device 100and define a rule of transmitting photo information and locationinformation upon receipt of an event (e.g., message) from at least onetarget user device 200. According to an exemplary embodiment of thepresent invention, the user (or parent of the user may generate a rulesuch as “take a photo and send the photo with my location if a text isreceived from mom in protection mode.” The rule may be generated throughspeech interaction with the microphone 143 or text interaction with theinput unit 120 or touchscreen 130 as to be described later. It ispreferred that the voice and text interaction is based on a naturallanguage as described above. In the rule defined as shown in FIG. 11,the situation to be detected, e.g., condition, specified in the definedrule may be “if a text is received from mom in protection mode,” and theaction to be taken in fulfillment of the condition may be “take a photoand send it with my location.” Also, an additional condition such as “ifa message including a phrase asking for location such as ‘where’ frommom” can be further configured.

The user device 100 may provide an interface for designating at leastone target user device 200 generating event and mapping the informationon at least one target device 200 (e.g., phone number, name, andnickname) to the rule. The rule may be predefined and input or redefinedby the user if necessary anytime anywhere, and the instruction forexecuting the rule may be input through a given interface in the form ofspeech, text, or gesture. Then the user device 100 may map theinstruction to the defined rule and store the mapping information.

In the state in which the rule has been defined, the user may executethe rule with the predefined instruction (e.g., speech, text, andgesture), whenever necessary. For example, the user may input thecorresponding instruction as a speech input (e.g., “protection mode”,“after school”, and “I will come home from school”) on the way home fromschool (institute) in the exemplary case of FIGS. 5A to 5E. Although thedescription is directed to the case in which the instruction is thespeech instruction input through the microphone 143, the instruction maybe input in the form of text by means of the input unit 120 ortouchscreen 130 or in the form of gesture. It is preferred that thespeech and text instructions are the natural language-based instructionsas described above. When the user intends to make a speech input, theuser may take a preliminary action (e.g., turning on the microphone 143)for notifying the user device 100 of the use of speech input forexecuting the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 may alsodetect the fulfillment of the condition (situation).

For example, the user device 100 may detect the fulfillment ofcondition1 such as “protection mode” specified in the defined rule andcheck condition3 such as receipt of text message (text message includingspecific condition such as “where”) from condition2 such as a designatedtarget device 200 according to conditon1. The user device 100 also mayperform action2 of acquiring the photo information through auto-shootingaccording to action1. The user device 100 also may perform action3 ofactivating the location positioning module 117. The user device 100 alsomay perform action4 of acquiring the location information on the userdevice 100. The user device 100 also may perform action5 of transmittingthe photo information and location information acquired through action1to action4 to the target user device 200.

If the location information transmitted by the user device 100 isreceived, the target user device 200 may perform feedback of ambientphoto information and location information on the user device through apredetermined interface. For example, the target user device 200 maydisplay the user device location on a map image as shown in FIG. 11.

Although not shown in FIG. 11, the photo information around the locationof the user device 100 may be acquired from an external server. Forexample, if the user device 100 captures a photo in the pocket of theuser, the photo may be blackout one other than an intended scene photo(e.g., ambient photo). Accordingly, the user device 100 has to analyzethe capture photo and check (e.g., determine) whether the user device100 is in a pocket based on the ambient condition of the user device 100(e.g., using a luminance sensor) to determine whether the photo is takenin the normal state. The user device 100 may be configured with theconditions and action to operate in such a way of acquiring the ambientphotos around the user device's current location from an external serverif the photo has been taken in an abnormal state.

FIG. 11 shows an exemplary case in which the owner of the user device100 inputs the instruction for executing the rule. However, a child usermay not execute the rule in dexterity. By taking notice such asituation, it is preferred to allow a parent to execute the ruleremotely to monitor the location of the child continuously. For example,configuring a rule such as “execute protection mode and inform of mylocation upon receipt of text message such as ‘where’ from mom” may bepossible. If a message including ‘where’ from a target user device 200,then the user device 100 checks (e.g., determines) whether the userdevice 100 operates in the protection mode and, if so, performs theabove-described operation. Otherwise, if the user device 100 is not inthe protection mode, the user device 100 executes the protection modeand performs the above-described operation.

As described above, the CAS service according to the exemplaryembodiment of FIG. 11 makes notifying one user (parent) of an eventtriggered by location and ambient environment of the other user (child)efficiently and quickly possible. The user (parent) may also acquire theinformation on the location, moving path, and ambient environment of theother user (child).

FIG. 12 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention.

FIG. 12 shows an exemplary situation in which the user configures andexecutes a rule through speech input such that the user device 100monitors to detect fulfillment of the condition specified in the ruleand executes a the corresponding action when the condition is fulfilled.As illustrated in FIG. 12, the user may define the rule of controlling aspecific function of the user device 100 in a specific environment. Theuser device 100 executes the rule according to the user input andexecutes the action of processing a specific function of the sure deviceas defined in the execution rule.

Referring to FIG. 12, the user may define a rule using the user device100. For example, the user may activate the function (or application)capable of generating a rule by manipulating the user device 100 anddefine the rule for executing plural functions (applications) in aspecific environment. According to an exemplary embodiment of thepresent invention, the user may generate a rule such as “turn on Wi-Fiand execute music APP if getting on a subway train.” The rule may begenerated by speech input through the microphone 143 or text inputthrough the input unit 120 or touchscreen 130. The speech and text inputmay be made with a natural language. In the case of FIG. 12, thesituation to be detected (e.g., condition), may be “if getting on asubway train” specified by the user explicitly, and the action to betaken in fulfillment of the condition may be “turn on Wi-Fi and executemusic APP.”

The user device 100 may provide the interface for selecting pluralactions (e.g., functions and applications) in fulfillment of thecorresponding function such that the user may select plural actions indefining the rule. For example, if the user enters “subway” as acondition of the rule, the user device 100 may display a list (actionlist) of actions (e.g., functions and applications) capable of beingexecuted in association with condition “subway” to receive a user inputfor selecting an action (e.g., Wi-Fi turn-on and music applicationexecution) to be executed from the action list. As described above, therule may be predefined or generated and redefined by the user in realtime, whenever necessary, anytime anywhere.

In the state in which the rule has been defined, the user may activateand/or execute the rule using the defined instruction (e.g., speech,text, and gesture) whenever necessary. For example, the user may make aninstruction (e.g., speech input such as “get on a subway train”,“subway”, and “subway mode”) before or upon getting on the subway trainto execute the predefined rule as in the exemplary case of FIGS. 5A to5E. Although the description is directed to the case in which theinstruction is the speech instruction input through the microphone 143,the instruction may be input in the form of text by means of the inputunit 120 or touchscreen 130 or in the form of gesture. It is preferredthat the speech and text instructions are the natural language-basedinstructions as described above. As described above, when the userintends to make a speech input, the user may take a preliminary action(e.g., turning on the microphone 143) for notifying the user device 100of the use of speech input for executing the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 may alsodetect the fulfillment of the condition (situation). For example, theuser device 100 may detect the fulfillment of condition1 such as “get ona subway train” specified in the defined rule and execute action1 ofturning on the Wi-Fi and action2 executing the music application. Theuser device 100 may process signals exchanged for establishing Wi-Ficonnection and playback of a music file as action1 and action2.

If the rule is executed and the condition specified in the rule isfulfilled (e.g., getting on a subway train), the user device 100 turnsof the Wi-Fi function and executes the music application with thefeedback of the execution result. For example, as shown in FIG. 12, theuser device 100 may display indication of Wi-Fi turn-on state and outputsound as a consequence of music playback.

According to the exemplary embodiment of FIG. 12, the user may make aspeech input or behavior input when the condition specified in thecurrently running rule is fulfilled to execute plural actions associatedwith the condition.

FIG. 13 is a diagram illustrating an exemplary situation of providingCAS using a user device according to an exemplary embodiment of thepresent invention.

FIG. 13 shows an exemplary situation in which the user configures andexecutes a rule through natural language-based speech or text input suchthat the user device 100 monitors to detect fulfillment of the conditionspecified in the rule and execute the corresponding action when thecondition is fulfilled. For example, when a rule is activated, the userdevice 100 monitors to detect a context corresponding to a condition ofthe activated rule. If the context is detected, the user device executesthe action corresponding to the condition of the rule. As illustrated inFIG. 13, the user may define an abstract condition such that the suredevice 100 executes the rule according to the user input and controlsprocessing specific function specified in the rule and communicationwith an external device (or an object capable of communication with theuser device 100 or to which a device capable of communicating the userdevice 100 is attached).

Referring to FIG. 13, the user may define the rule using the user device100. For example, the user may activate the function (or application)capable of generating a rule by manipulating the user device 100 anddefine the rule for executing plural functions (user device's functionsand external device's function) specified to be executed in a specificenvironment.

According to an exemplary embodiment of the present invention, the usermay generate a rule such as “adjust lamp brightness to level 2 and playclassical music if gloomy.” At this time, the rule may be generated byspeech input through the microphone 143 or text input through the inputunit 120 or touchscreen 130. The speech and text input may be made witha natural language. In the case of FIG. 13, the situation to be detected(e.g., condition), may be “if gloomy” and the action to be taken infulfillment of the condition may be “adjust lamp brightness to level 2and play classical music if gloomy.”

At this time, the user device 100 may provide an interface for the userto select plural actions (e.g., user device's function control andexternal device function control) for execution in determining the ruleaccording to the sure request. For example, if the user inputs “gloomy”as the condition specified in the rule, the user device 100 may displaya list of actions executable in association with the condition “gloomy”to prompt the user to select an action (e.g., lamp brightness control,music application execution, and classic music playback) from the actionlist. The rule may be predefined or generated and redefined by the userin real time, whenever necessary, anytime, anywhere.

In the state in which the rule has been defined, the user may executethe rule using the defined instruction (e.g., speech, text, and gesture)whenever necessary. For example, the user may make an instruction (e.g.,speech input such as “gloomy”, “tired”, and “gloomy mode”) to executethe predefined rule as in the exemplary case of FIGS. 5A to 5E. Althoughthe description is directed to the case in which the instruction is thespeech instruction input through the microphone 143, the instruction maybe input in the form of text by means of the input unit 120 ortouchscreen 130 or in the form of gesture. It is preferred that thespeech and text instructions are the natural language-based instructionsas described above. As described above, when the user intends to make aspeech input, the user may take a preliminary action (e.g., turning onthe microphone 143 with function key or execution button of a widget)for notifying the user device 100 of the use of speech input forexecuting the rule in advance.

The user device 100 may recognize and parse the speech input to executethe rule indicated by the speech input. The user device 100 may alsodetect the fulfillment of the condition (situation). For example, theuser device 100 may detect the fulfillment of condition such as “gloomy”specified in the defined rule and execute action1 of adjusting the lampbrightness to level 2 by controlling an external device and action2executing the music application to perform complete action3 of playingclassical music.

In the exemplary case of FIG. 13, if the external device (e.g., theliving room lamp or other external device to which the user device 100has permission to communicate and/or control) is capable ofcommunicating with the user device 100, the external device maycommunicate with the user device 100 and the external device may becontrolled directly by the user device 100 and, otherwise, indirectlyusing an auxiliary device (e.g., power control device havingcommunication). According to an exemplary embodiment of the presentinvention, the user device 100 may communicate with the power controldevice capable of communication to adjust the brightness of the livingroom lamp.

If an instruction for executing the rule, the user device 100 may adjustthe external lamp brightness and execute the music applicationsimultaneously and feeds back the execution result. For example, asshown in FIG. 13, the user device 100 may adjust the brightness of thelight lamp to level 2 and output the sound as a consequence of classicalmusic playback and corresponding execution screen.

As described above, according to the exemplary embodiment of FIG. 13,the user may define an abstract condition and configure a rule toexecute an action depending on the situation. According to an exemplaryembodiment of the present invention, the user may define the user'scondition such as “gloomy (or tired)” based on a natural language andprovide feedback with the execution of an action in fulfillment of thecondition.

Hereinabove, FIGS. 6 to 13 have been directed to the case in which therule and instruction are defined and configured separately. However,according to various exemplary embodiments of the present invention, theinstruction for executing the rule may be extracted from the predefinedrule without extra definition.

Suppose that a rule such as “if I get in a taxi, send my location tofather and little brother (sister) at every 5 minutes” is defined as inthe exemplary case of FIG. 7. If the user device 100 recognizes “if Iget in a taxi” in the rule, the user device 100 may extract associativeinstructions (or associative word or command having high relationshipwith the words in the rule defined as commands for execution of therule) such as “take taxi now,” “taxi,” and “taxi mode.” For example, theuser may execute the rule just by inputting an associative command suchas “take taxi now,” “taxi,” and “taxi mode” in the state in which therule has been defined without designation of a certain command.

Suppose that a rule such as “if a text is received from wife at thewheel, send my current location information” is defined as in theexemplary case of FIG. 8. The user device 100 recognizes “at the wheel”in the rule and extracts associative commands of “I will drive,”“drive,” and “driving move” to execute the rule. For example, the usermay execute the rule with the input of any of the associative commands“I will drive,” “drive,” and “driving move” in the state in which therule has been defined without designation of a certain command.

If any associative command is input, the user device 100 may search thepredefined rules for anyone matching the command to execute thecorresponding rule.

The operations described with reference to FIGS. 7 and 8 may be appliedto the exemplary cases of FIGS. 6 and 9 to 13. For example, anassociative command such as “I will drive,” “drive,” and “driving move”may be input in association with the rule defined in the situation ofFIG. 9; an associative command such as “medicine,” “check medicinebottle whereabouts,” and “medication time” may be input in associationwith the rule defined in the situation of FIG. 10; an associativecommand such as “protection mode,” “after school,” and “I will go home”may be input in association with the rule defined in the situation ofFIG. 11; an associative command such as “get on subway train,” “subway,”or “subway mode” may be input in association with the rule defined inthe situation of FIG. 12; and an associative command such as “gloomy”and “tired” may be input in association with the rule defined in thesituation of FIG. 13. Accordingly, the user device 100 may execute thecorresponding rule based on the associative command corresponding to thepredefined rule without defining ay extra command.

FIGS. 14A and 14B are diagrams illustrating exemplary screens forexplaining operations of stopping a currently running rule temporarilyin a user device according to an exemplary embodiment of the presentinvention.

FIG. 14A shows an exemplary screen of the user device 100 in which a CASwidget 500 is provided with an execution information region 520 listingthe currently running rules.

FIG. 14A is directed to an exemplary case in which “taxi” and “subway”rules are running currently. The user may stop a certain rule byselecting the rule from the execution information region 520 of thewidget 500. For example, in the state of FIG. 14a , the user may select(e.g., make touch gesture of tap) the “taxi” rule. Then the user device100 stops the rule (e.g., taxi) temporarily in response to the user'sselection among the currently running rules (e.g., taxi and subway) asshown in FIG. 14B.

In this case, the user device 100 may change a marking to the rulestopped temporarily. According to an exemplary embodiment of the presentinvention, the user device 100 may change the state indication mark ofthe rule of “taxi” from the enabled state mark to the disabled statemark as shown in FIG. 14B. For example, each of the rules listed in theexecution information region 520 of the widget 500 is provide with astate indication button indicating whether the corresponding rule isenabled or disabled.

According to an exemplary embodiment of the present invention, thewidget 500 may indicate the currently running rules and may stop eachrule temporarily according to the user input. Accordingly, stopping theoperation of a rule taking a repetitive action (e.g., action checking acondition or sending a text message periodically) whenever necessaryaccording to the user's intention so as to improve usability ispossible.

FIGS. 15A and 15B are diagrams illustrating exemplary screens forexplaining operations of stopping a currently running rule temporarilyin a user device according to an exemplary embodiment of the presentinvention.

FIGS. 15A and 15B shows the screens of the user device 100 when the CASapplication (e.g., rule list) is executed according to an exemplaryembodiment of the present invention.

Referring to FIGS. 15A and 15B, the rule “subway” is currently running,the detailed information specified in association with the selected rulepresented in a full-down window. For example, the user may select therule item 1510 to check the detailed information of the correspondingrule (e.g., conditions and actions) 1520 and 1530, e.g., “Wi-Fi setting:On” and “sound setting: vibration”

In the state of FIG. 15A, the user may select one of the plural actionitems 1520 and 1530 specified in one rule to stop running temporarily.For example, the user may select the “Wi-Fi setting” item 1520 among theactions, e.g., Wi-Fi setting item 1520 and sound setting item 1530, ofthe rule “subway” 1510. Then the user device 100 stops the action (e.g.,the Wi-Fi setting) corresponding to the item selected by the user amongthe actions of the currently running rule (e.g., “subway”) 1510.

The user device 100 may change the state indication mark of the stoppedaction. For example, the user device 100 may change the state indicationmark of the action of “Wi-Fi setting” from the enabled state mark to thedisabled state mark as shown in FIG. 15B. For example, each of theactions 1520 and 1530 of the rule is provide with a state indicationbutton indicating whether the corresponding action is enabled ordisabled.

According to an exemplary embodiment of the present invention, whenplural actions are running in association with one rule, stopping therespective actions listed in the current running action list selectivelyis possible.

FIGS. 16A to 16C are diagrams illustrating exemplary screens forexplaining operations of stopping currently running rule in a userdevice according to an exemplary embodiment of the present invention.

FIG. 16A shows an exemplary screen of the user device 100 which isdisplayed in the state where the CAS widget 500 is executed with theexecution information region 520 presenting information on the currentlyrunning rules.

FIG. 16A is directed to an exemplary case in which the “taxi” and“subway” rules are currently running. The user may select (e.g. make atouch gesture of tap) an instruction input region (or rule executionbutton) 510 of the widget 500. Then the control unit 170 determinesselection of the instruction input region 510 as the initiation of oneof rule execution, temporary stop of the currently running rule, andtermination of the currently running rule. FIGS. 16A to 16C are directedto an exemplary case of stopping the currently running rule temporarily.As shown in FIG. 16B, the user device 100 may displays a popup window1651 prompting the user to input an instruction (e.g., speech input) forstopping a rule temporarily with a message (e.g., “please speakcommand”), and waits for user's speech.

In the state of FIG. 16B, the user may make a speech input (e.g., “stoptaxi temporarily” for stopping a target rule (e.g., taxi). Then thecontrol unit 170 may recognize the user's speech input and stop the rule(e.g., “taxi”) corresponding to the speech among the currently runningrules (e.g., “taxi” and “subway”). As shown in FIG. 16C, the controlunit 170 may change the state indication mark of the rule (e.g., “taxi”)selected in the execution information region 520 of the widget 500 bythe speech input so as to be stopped temporarily.

According to an exemplary embodiment of the present invention, thecontrol unit 170 may change the state indication mark of the “taxi” rulefrom the enabled state mark to the disable state mark in the executioninformation region 520 as shown in FIG. 16C. For example, FIG. 16C showsan exemplary screen of the user device 100 in the state in which aspecific rule (e.g., taxi) is stopped temporarily in the executioninformation region 520 of the widget 500.

According to an exemplary embodiment of the present invention, thecurrently running rules are presented by means of the widget 500 andstopped temporarily according to the user's speech input (e.g. “stop OOOtemporarily” where OOO may be a rule or a condition).

FIG. 17 is a diagram illustrating an exemplary screen with an indicationof an execution rule in a user device according to an exemplaryembodiment of the present invention.

Referring to FIG. 17, when a rule is executed in response to the userrequest, the user device 100 may display a notification item 700 on thescreen of the display panel 131. The notification item 700 may beprovided in the form of an icon or text representing the executed rule.For example, if a driving-related rule is executed, the notificationitem 700 in the form of a car image (or text) may be provided at a partof the screen. Also, if a medicine-related rule is executed, thenotification item 700 in the form of a medicine or medicine bottle image(or text) is provided at a part of the screen. According to variousexemplary embodiments of the present invention, the currently runningrule is notified to the user by means of the notification item 700 suchthat the user is aware of the currently running rule intuitively.

FIG. 17 is directed to an exemplary case in which the notification item700 is provided at a part on the display panel 131. However, exemplaryembodiments of the present invention are not limited thereto. Thenotification may be presented in the indicator region for providingvarious operation state information of the user device 100 as describedwith reference to FIG. 5E.

FIGS. 18A and 18B are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention.

Referring to FIGS. 18A and 18B, the user device 100 may provide anotification item 550 at a part of the indicator region 1850 on thescreen of the display panel 131. According to an exemplary embodiment ofthe present invention, the notification item 550 is presented at a leftpart of the indicator region 1850 to inform of the currently runningrule.

Referring to FIGS. 18A and 18B, in various exemplary embodiments of thepresent invention, the notification item 550 may be presented when atleast one rule is running currently.

Referring to the exemplary case of FIG. 18A, multiple rules (e.g., tworules of “taxi” and “subway”) are running as shown in the executioninformation region 520 of the widget 500. Referring to FIG. 18B, one(e.g., “subway”) of the rules (e.g., “taxi” and “subway” rules) listedin the execution information region 520 is enabled while the other rule(e.g., “taxi”) is disabled (e.g., stopped temporarily as described inthe exemplary embodiments).

According to various exemplary embodiments of the present invention, thedevice 100 may notify the user of the existence of at least on currentlyrunning rule using the notification item 550. However, exemplaryembodiments of the present invention are not limited thereto. Thenotification item may be provided per rule. This means that multiplenotification items may be presented in the indicator region 1850 inmatch with the number of the currently running items.

FIGS. 19A and 19B are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention.

Referring to FIGS. 19A and 19B, when a rule is executed in response tothe user request, the user device 100 may display a notification item550 informing of the existence of current running rule(s) at theindicator region 1850 on the screen of the display panel 131. Accordingto an exemplary embodiment of the present invention, if the currentlyrunning rule stops temporarily, the user device 100 may notifying of theexistence of temporarily stopped rule using the correspondingnotification item 550 of the indicator region 1850.

As shown in the exemplary case of FIG. 19A, if the currently runningrule (e.g., taxi) is stopped temporarily by the user selection, the userdevice 100 switching the state of the notification item 550 to disabledstate. The execution information region 520 of the widget 500 indicatesthat the tax rule is not currently running.

According to an exemplary embodiment of the present invention, as shownin FIG. 19B, the notification item 550 may be provided in the form of anicon representing the corresponding rule or an application executionicon. The icon may be presented in one of the enabled state icon anddisabled state icon for intuitive indication.

FIGS. 20A to 20C are diagrams illustrating exemplary screens with anitem notifying of the rule(s) executed in a user device according to anexemplary embodiment of the present invention.

Referring to FIGS. 20A to 20C, the user device 100 may provide thenotification item 550 for at least one execution rule in the indicatorregion 1850. According to an exemplary embodiment of the presentinvention, the user may select (e.g., tap, make a touch gesture, and thelike) the notification item 550 of the indicator region 1850 or touchand drag the indicator region 1850 downward to display a quick panel2010.

The quick panel 2010 may be configured to display settings of variousfunctions (e.g., Wi-Fi, GPS, sound, screen rotation, power saving mode,and the like) and configure the settings quickly in the form of asemi-transparent window sliding in response to the user input to overlaythe screen of the display entirely or partially. In various exemplaryembodiments of the present invention, the quick panel 2010 may beprovided with the information item(s) 2050 representing thecorresponding rule executed currently as shown in FIG. 20B. Accordingly,the user is capable of checking the condition and action of the at leastone rule represented by the notification item 550 presented at theindicator region 1850 intuitively through the quick panel 2010.

In the state of FIG. 20B, the user may select (e.g., tap, make a touchgesture, and the like) an information item 2050 of the quick panel 2010to check the detailed information on the corresponding rule as shown inFIG. 20C. In response to the selection of the information item 2050, theuser device 100 may display the detailed information corresponding tothe currently running rule (represented by the information item selectedby the user). In various exemplary embodiments of the present invention,the detailed information may be provided in the form of a text popupwindow presented on the quick panel 2050, a rule list through screenswitching as described above, or speech output in the state ofmaintaining the quick panel 2050.

According to various exemplary embodiments of the present invention, ifthe notification item 550 of the indicator region 1850 is touched, theuser device 100 feeds back the conditions and actions specified in therule in the form of speech or text output.

According to various exemplary embodiments, if the notification item 550of the indicator region 1850 is touched or touched and dragged, thequick panel 2010 is displayed to show the information item 2050representing the corresponding rule. If the information item 205 istouched, the detailed information (e.g., condition and action) on thecorresponding rule may be fed back in the form of a speech output.According to an exemplary embodiment of the present invention, if theinformation item 2050 is selected as shown in FIG. 20C, a speech of“[home] is running” may be output. Of course, various notifications suchas “ringtone is a bell sound” and “Wi-Fi ON” may be output in the formof at least one of speech and text.

FIGS. 21A and 21B are diagrams illustrating exemplary screens associatedwith an operation of notifying of an execution rule in a user deviceaccording to an exemplary embodiment of the present invention.

FIGS. 21A and 21B show an exemplary operation of notifying the user ofthe currently running rules using a widget 500 according to an exemplaryembodiment of the present invention.

Referring to FIG. 21A, in the state in which the currently running rule(e.g., home) is displayed in the execution information region 520 of thewidget 500, the user may select the corresponding rule. Then the userdevice 100 may feed back the detailed information on the rule selectedin the execution information region 520 in the form of a text or speechfeedback.

For example, as shown in FIG. 21B, the user device 100 may output thecondition and action such as “[home] is running, ringtone is bell sound,Wi-Fi on” configured in association with the rule (e.g. home) selectedby the user in the form of a text popup window 2150. The user device 100also may output the detailed information according to the settingconfigured by the user in the form of speech feedback or both the speechand text feedbacks.

FIGS. 22A to 22C are diagrams illustrating exemplary screens associatedwith an operation of terminating a currently running rule in a userdevice according to an exemplary embodiment of the present invention.

FIG. 22A shows an exemplary screen of the user device 100 in the statein which the CAS widget 500 is executed which shows the information onthe currently running rules in the execution information region 520 ofthe widget 500 according to an exemplary embodiment of the presentinvention.

FIG. 22A is directed to the case in which the currently running rule isof “subway.” In this case, the information on the “subway” is displayedin the execution information region of the widget 500 along with thenotification item 550 at the indicator region 1580 indicating that therule “subway” is running currently.

The user may select (e.g., make a touch gesture of tap) the instructioninput region (or rule execution button) 510. Then the control unit 170determines that the selection in the instruction input region 510corresponds to the initiation of one of the rule execution, thetemporary stop of the currently running rule, and the termination of thecurrently running rule. FIGS. 22A to 22C are directed to an exemplarycase of terminating the currently running rule. As shown in FIG. 22B,the user device 100 may display a popup window 2251 prompting the userto input an instruction (e.g., speech input) for terminating the rulewith a guidance message (e.g., “please speak command”) and waits foruser's speech.

In the state of FIG. 22B, the user may make a speech input (e.g. “endsubway”) for terminating execution of the rule (e.g., subway). Then thecontrol unit 170 may recognize the speech input and may terminate therule indicated by the speech input (e.g., “subway”). The control unit170 may change the display of the execution information region 520 ofthe widget 500 by reflecting the termination of the rule (e.g.,“subway”) as shown in FIG. 22C. The control unit 170 may control suchthat the notification item 550 disappears in the indication region 1850by reflecting the termination of the rule “subway.”

According to an exemplary embodiment of the present invention, thecontrol unit 170 may change the display of the state of the rule“subway” from the enabled state to the disabled state in the executioninformation region 520 as shown in FIG. 22C. FIG. 22C shows theexemplary screen of the user device 100 in which the rule (e.g. subway)has disappeared in the execution information region 520 of the widget500 according to the user's speech input for termination of thecorresponding rule. As the rule “subway” is terminated, the informationitem related to the rule “subway” is replaced by a notification comment“no running rule.” In the case in which one of the currently runningrules is terminated, only the information item corresponding to theterminated rule disappears while the information items corresponding tothe other currently running rules are maintained in the executioninformation region 520.

According to various exemplary embodiments of the present invention, acurrently rule may be terminated in response to the user's speech inputcommanding termination of the rule in the form of “terminate 000.” Invarious exemplary embodiments of the present invention, if a specificrule is terminated in response to the user input, at least one settingconfigured in association with the condition and action of the rule maybe recovered to the state before the execution of the corresponding ruleautomatically.

FIGS. 23A and 23B are diagrams illustrating exemplary screens associatedwith the operation of terminating the currently running rule in the userdevice according to an exemplary embodiment of the present invention.

FIGS. 23A and 23B show an exemplary operation of providing an end buttonfor terminating a rule represented by an item in a widget 500 or a rulelist and terminating the corresponding rule using the end button.

As shown in FIG. 23A, the widget 500 shows that a rule of “subway” iscurrently running. As shown in FIG. 23A, a notification item 550 isprovided at the indicator region 1850 to indicate the existence of anycurrently running rule.

The user may terminate the operation of the currently running rule usingthe end button 525 mapped to the corresponding rule prior to thetermination of the rule “subway.” For example, the user may select(e.g., tap or touch gesture) the information item representing the rule(e.g., “subway”) in the execution information region 520 of the widget500 as shown in FIG. 23A.

Then the control unit 170 recognizes the user input made to the endbutton 525 and terminates the rule (e.g., “subway”) corresponding to theend button 525. As shown in FIG. 23B, the control unit 170 may controlsuch that the item representing the rule (e.g., “subway”) terminated bythe user disappears in the execution information region 520. The controlunit 170 also may control such that the notification item 550 disappearsin the indicator region 1850 as a result of the termination of the rule.

According to an exemplary embodiment, the control unit 170 may changethe execution state of the rule “subway” from the enabled state to thedisabled state in the execution information region 520 as shown in FIG.23B. FIG. 23B shows the exemplary screen of the user device 100 in thestate in which the item indicating the enabled state of the rule (e.g.,subway) has disappeared from the execution information region of thewidget in response to the selection of the end button 525. In responseto the request for terminating the currently running rule “subway”, theinformation item indicating the enabled state of the rule “subway” isreplaced by a notification item “not running rule.” Assuming that one ofthe plural running rules is terminated, only the information item of theterminated rule disappears while the information items corresponding tothe other currently running rules are maintained in the executioninformation region 520.

FIGS. 24 and 25 are diagrams illustrating a situation of terminating CASservice in a user device according to an exemplary embodiment of thepresent invention.

Referring to FIG. 24, the user may make an instruction (e.g., a speech,a text, a selection of an end button of widget, and a gesture)predefined for terminating a rule to end one of the currently runningrules. The rule termination instruction may be a speech instruction or atext instruction. The rule may be terminated by means of apre-designated function key, a button designated pre rule in the widget500, a text-scribing input, or predefined gesture.

For example, as shown in FIG. 24, the user may make a speech input suchas “end 000 (command corresponding to the rule),” “arrived home,” and“arrived home almost” based on a natural language. Then the user device100 may recognize and parse the speech input and ends the rule, if thespeech input matches the predetermined instruction. In various exemplaryembodiments of the present invention, the rule termination instructionmay be set per rule. The rule termination instruction may be set as acommon instruction for all of the rules. In the case of the common rule,making the termination instruction to end all of the currently runningrules at a time is possible.

Referring to FIG. 25, although any instruction for terminating acurrently running rule is not input by the user, the user device 100terminates the rule or prompts the user to terminate the rule when aspecific condition is fulfilled.

For example, as shown in FIG. 25, the user device 100 may monitor asituation of the currently running rule and determine whether thesituation fulfills the rule termination condition registered with amapping table. In various exemplary embodiments of the presentinvention, the rule termination condition may be fulfilled when nochange in situation is detected for a predetermined duration or a ruletermination condition specified by the user (e.g., speech input, ruleterminal function key selection, and maintain a specific condition forterminating a rule) is satisfied.

When the rule termination condition is fulfilled, the user device 100may display a popup window 900 presenting a message prompting the userto terminate the corresponding rule (e.g., “terminate the drivingmode?”) and maintain or terminate the corresponding rule depending onthe user's interaction to the popup window 900.

FIGS. 26A and 26B are diagrams illustrating exemplary screens associatedwith an operation of deleting a rule in a user device according to anexemplary embodiment of the present invention.

FIGS. 26A and 26B show the exemplary screens of the user device 100 whenthe CAS application (e.g., rule list) is executed.

FIGS. 26A and 26B are directed to an exemplary case in which three rules“home,” “taxi,” and “subway” defined by the user are stored in the userdevice 100. In the state in which the three rules are presented in therule list, the user may make a delete instruction through menumanipulation, speech input, or gesture input.

For example, the user may request for execution of the delete functionby manipulating the menu of the user device 100, making a speech or textinput of “delete rule”, or making a gesture input, in the state of FIG.26A. Then the control unit 170 activates the rule deletion function inresponse to the user input with the display of the screen interfacecapable of deleting the rule as shown in FIG. 26B. According to anexemplary embodiment of the present invention, a list of predeterminedrules may be displayed along with selection item 2600 (e.g. check box)per rule as shown in FIG. 26B.

The user may select at least one rule by check the correspondingselection item. At this time, the control unit 170 may check theselection item 2600 with a mark to indicate that the corresponding ruleis selected. According to an exemplary embodiment, the uncheckedselection item 2600 is presented as an empty box (e.g., a selection boxwithout a checkmark therein) while the checked selection item 2600 ispresented as a box with a checkmark therein. The user may delete atleast one selected rule through menu manipulation or with a deletionbutton.

Although FIGS. 26A and 26B are directed to an exemplary case of deletinga rule using the menu and selection item 2600, the user may delete therule by inputting a speech or text instruction. For example, in thestate of FIGS. 26A and 26B, the user may make a speech or scribing inputof “delete home” to delete the rule “home.”

FIG. 27 is a flowchart illustrating a procedure of generating a rule ina user device according to an exemplary embodiment of the presentinvention.

Referring to FIG. 27, the control unit 170 (e.g., rule configurationmodule 173) receives a user input requesting for CAS at step 1001. Forexample, the user may execute the configuration mode for defining rulesof CAS by means of one of the input unit 120, microphone 143, andtouchscreen 130. In the configuration mode, the user may make an inputfor defining a rule in the form of speech input through the microphoneor text input through the input unit 120 or touchscreen 130. Then thecontrol unit 170 may recognize the user input for CAS in theconfiguration mode. The user may input instructions associated with therule, condition (or situation), and action through one of theaforementioned input means.

For example, the user may define a rule of “send text if getting intaxi,” designate at least one target user device (e.g., “to father”) tosend the text, and define a rule execution instruction of “I got in ataxi.” As described in the above exemplary embodiments of the presentinvention, defining a detailed rule such as “if I get in a taxi, send mylocation to father and little brother (sister) at every 5 minutes” or“if a text message including ‘where’ is received from father, send atext message to father” is possible.

The control unit 170 recognizes the user input at step 1003. Forexample, the control unit 170 may recognize the speech or text inputthrough a corresponding input means in the rule configuration mode. Forexample, the control unit 170 performs the speech recognition functionfor recognizing a speech input through the microphone 143 or textrecognition function for recognizing a text input through the input unit120 or the touchscreen 130. It may be preferred that the speech and textinstruction is made based on a natural language as described above.

The control unit 170 parses the recognized user input (e.g., naturallanguage-based speech or text) at step 1005. For example, the controlunit 170 parses the natural language-based speech instruction to extractand identify the user-intended rule, condition, and rule executioncommand. The rule, condition (situation), action, and rule executioninstruction may be input in series according to guidance. The controlunit 170 also may search for the item (e.g., situation to be detected,target, and command) of executing an action according to the recognizedsituation as a result of parsing the user input to check the missingpart. According to an exemplary embodiment of the present invention, thecontrol unit 170 may generate a rule based on the interaction with theuser by providing guidance for generating the rule and receiving theinformation according to the guidance.

For example, when the user defines the rule for situation recognitionsuch as “send text if I get in a taxi” without designation of a target,the control unit 170 may recognize the absence of the destination (e.g.,target) of the text message. When the action for the rule definedwithout designation of the target is executed, the control unit 170 mayguide to designate a target. For example, the control unit 170 mayperform speech or text interaction with the user through to acquireadditional information according to the action. According to anexemplary embodiment of the present invention, the control unit 170 maydisplay a popup text (with speech guidance) such as “please speakrecipient” or “send whereto.” Then the user may make a speech input suchas “designate later,” “boyfriend” and “send to father.” The target maybe designated through natural language-based speech or text input asdescribed above. The rule is complemented with additional information;the control unit 170 may perform recognition and parsing on the speechinput and match the corresponding items to one rule according to theabove described procedure.

The control unit 170 manages the rules defined for CAS based on theparsed user input at step 1007. For example, the control unit 170 maymap the rule, condition (situation), action, and rule executioninstruction acquired by parsing the user input among each other andstores the mappings in a mapping table for management.

Although not depicted in FIG. 27, user guidance may be provided indefining the rule according to various exemplary embodiment of thepresent invention. For example, the user may input the instruction forgenerating a rule by typing with a virtual key pad or writing on thetouchscreen with a specific input tool (e.g., electronic pen and user'sfinger). Then the control unit 170 may provide a screen interfacecapable of recognizing the text input through writing or keypad-typingand defining the rule based on the recognized text.

The screen interface may provide the lists (e.g., a condition list andan action list) of the conditions and actions (or functions orapplications), and the user may selectively turn on/off the conditionsand actions (or functions or applications). For example, if the userintends to define a rule for turning off the GPS function at office, theuser may input “office” on the screen of the user device 100 by means ofan electronic pen (scribing input or touch keypad-based characterselection input).

According to various exemplary embodiments of the present invention, theuser device 100 may provide the screen interface (e.g., command pad andtouch pad) capable of receiving the scribing input or typing input. Theuser device 100 may recognize text “office” input through the scribingor typing made on the screen interface such as command pad and touch padby means of an input tool (electronic pen or user's finger). Then thecontrol unit 170 controls to display a configuration list screen suchthat the user turns on/off at least one of the conditions and actions(functions or applications) associated with “office” on theconfiguration list screen.

According to an exemplary embodiment of the present invention, the rulemay be defined through one of the speech or text input functions. Thetext input may be made by writing/typing natural language instruction asdescribed above or entering a keyword (e.g. “office”) by means of thecommand pad or the touch pad and then turning on/off at least one of theactions presented in a list.

FIG. 28 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention.

FIG. 28 shows the operation procedure of executing a rule, at the userdevice 100, in response to a user interaction and taking an action whena condition (situation) specified in the rule is fulfilled.

Referring to FIG. 28, the control unit 170 receives a user instructionfor CAS at step 1101. For example, the user may make a speech input forexecuting a rule defined for CAS through the microphone 143. Then thecontrol unit 170 may receive the user's speech instruction through themicrophone 143. According to various exemplary embodiments of thepresent invention, the instruction for executing the rule may be inputin the form of speech input through the microphone 143, text inputthrough the input unit 120 or the touchscreen 130, or gesture input.

According to various exemplary embodiments of the present invention,designating one of the functions keys of the user device 100 as aninstruction key (e.g., shortcut key or rule execution button of thewidget 500) to stand by for speech input of the user may be possible. Inthis case, when the instruction key is selected, the control unit 170stands by for a speech input of the user for CAS and attempts speechrecognition for the speech input made in the standby state. The standbystate may be configured to be maintained after the standby mode commandor only while the instruction key is pressed.

The control unit 170 recognizes the instruction input by the user atstep 1103. For example, the control unit 170 may extract the commandinstructing execution of the rule from the user's speech input.

The control unit 170 parses the recognized instruction at step 1105. Forexample, the control unit 170 may parse the recognized user speech(e.g., “I get in a taxi”) to extract the rule execution command (e.g.,“taxi”).

The control unit 170 determines whether there is any rule matching theextracted execution command among the predetermined rules at step 1107.

If there is no rule matching the execution command at step 1107, thecontrol unit 170 controls to display guidance at step 1109. For example,the control unit 170 may display a popup guidance notifying that thereis not rule request by the user. The control unit 170 also may displayguidance asking whether to define the rule associated with thecorresponding command in the form of guidance popup. The control unit170 also may provide a list of the rules defined by the user.

The control unit 170 controls to perform the operation according to theuser request at step 1111 after the display of the guidance. Forexample, the control unit 170 may terminate the rule according to theuser selection, define a new rule associated with the command accordingto the user selection, or process the operation for selecting a specificrule from the rule list.

If a rule matches the execution command at step 1107, the control unit170 determines whether the number of rules matching the executioncommand is greater than 1 at step 1113. For example, the user may defineone or more rules matching one instruction. According to an exemplaryembodiment, the user may define a plurality of rules (e.g. a first ruleof “if an event is received from a designated external device at thewheel, send current location,” a second rule of “if the driving speed isequal to or greater than 100 Km/h, output alarm,” and “if the drivingspeed is equal to or greater than 60 Km/h, increase radio volume”). Asan example, the user may define the first rule with the command“driving1,” the second rule with the command “driving2,” and the thirdrule with the command “driving3.”

If the number of the rules matching the execution command is not greaterthan 1 (e.g., if only one rule matches the execution command), thecontrol unit 170 may control to execute the action according to thesingle rule at step 1115. For example, the control unit 170 monitors todetect whether the condition (situation) specified in the correspondingrule is fulfilled and executes, when the condition (situation) isfulfilled, one or more actions.

If the number of the rules matching the execution command is greaterthan 1 at step 1113, the control unit 170 controls to execute theactions corresponding to the plural rules matching the execution commandat step 1117. For example, the control unit 170 may monitor theconditions (situations) specified in the plural rules matching theexecution command and execute, whenever at least one condition isfulfilled, the action of every rule of which condition is fulfilled.

Although not depicted in FIG. 28, the control unit 170 may provide aninterface prompting to (recommending) one of the plural rules matchingone instruction. The control unit 170 may control to execute pluralactions according to the plural rules selected by the user (by selectioninput through speech, text, or gesture instruction) or a single actionaccording to a single-selected rule.

Although not depicted in FIG. 28, the control unit 170 may check theitems (e.g., condition (e.g., condition (situation) to be recognized ortarget)) for execution of an action in the rule according to the userinput to check a missing part. The control unit 170 may check theinteraction of the user in the process of executing the rule.

For example, when the user defines the rule for situation recognitionsuch as “send text if I get in a taxi” without designation of a target,the control unit 170 may recognize the absence of the destination (e.g.,target) of the text message. In this case, the control unit 170 mayrecognize the absence of the destination when preparing to execution theaction of transmitting a text message upon detection of the fulfillmentof the corresponding condition. When the action for the rule definedwithout designation of the target is executed, the control unit 170 mayguide to designate a target. According to an exemplary embodiment of thepresent invention, the control unit 170 may display a popup text (withspeech guidance) such as “whom to send the text?” Then the user maydesignate the target to which the text message is addressed such as“send to father” through natural language-based speech or text input.

FIG. 29 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention.

Particularly, FIG. 29 shows an exemplary procedure of executing a rule,checking one or more conditions (situations) specified in the rule toexecute an action or release the currently running rule.

Referring to FIG. 29, the control unit 170 executes a rule according tothe user request at step 1201. Next, the control unit 170 feeds back theexecution information (e.g., notification item) as a consequence of theexecution of the rule at step 1203. For example, the control unit 170may control to display the item (icon or text) associated with the ruleexecuted on the screen of the display panel 131 as described above.

The control unit 170 checks the condition specified in the rule at step1205. For example, the control unit 170 monitors continuously orperiodically to detect whether the condition (situation) specified inthe currently running rule is fulfilled.

The control unit 170 determines whether the action execution conditionis fulfilled based on the check result at step 1207. For example, thecontrol unit 170 monitors at least one condition (situation) specifiedin the currently running rule to determine whether the currentlysituation matches the specific condition of execution the actionspecified in the rule by referencing the mapping table.

If the condition (situation) of the user device 100 matches the actionexecution condition at step 1207, the control unit 170 controlsexecution of the action triggered by the fulfillment of the condition atstep 1209. For example, the control unit 170 monitors the condition orsituation and, if the condition or situation matches an action executioncondition, executes the corresponding action. The action may be ofexecuting a function (or application specified in the rule, generatingexecution result (e.g. context information), and output the executionresult to the user or others.

If the condition (situation) of the user device 100 does not match theaction execution condition at step 1207, the control unit 170 determineswhether the condition (situation) of the user device 100 matches therule release condition at step 1211. For example, the control unit 170monitors a certain condition specified in the currently running rule anddetermines whether the current situation matches the rule releasecondition specified in the rule by referencing the mapping table. Therule release condition may be fulfilled when there is no change insituation for the duration configured by the user or a user-specifiedrule release condition (e.g., rule release speech instruction, functionkey input, text input, gesture input, and continuation of specificcondition) is satisfied.

If the condition (situation) of the user device 100 does not the rulerelease condition at step 1211, the control unit 170 returns theprocedure to step 1205 to continue checking the condition of the userdevice.

If the condition (situation) of the user device 100 matches the rulerelease condition at step 1211, the control unit 170 release thecurrently running rule at step 1213 and feeds back the rule releaseinformation at step 1215. For example, the control unit 170 may controlto output at least one of audio, video, and tactile feedback. Accordingto an exemplary embodiment of the present invention, the control unit170 may control to output the rule release feedback in the form of atleast one of audio alarm (e.g., speech and sound effect), popup window,and vibration.

Although not depicted in FIG. 29, the control unit 170 may present apopup message prompting the user to terminate the executed rule when therule release condition is fulfilled to maintain or release the currentlyrunning rule depending on the user's selection.

FIG. 30 is a flowchart illustrating a procedure of providing CAS in auser device according to an exemplary embodiment of the presentinvention.

Referring to FIG. 30, the control unit 170 monitors to detect an eventin the state in which the rule executed in response to a user request isrunning at step 1301, and determines whether the event is detected atstep 1303. For example, the control unit 170 may monitor to detect aninternal or external event (condition) specified in association with thecurrently running rule.

The event (condition) may include an internal event occurring in theuser device 100 as a result of a change in internal condition and anexternal event received from outside. According to an exemplaryembodiment of the present invention, the internal event may include anevent occurring when the movement speed of the user device 100 is fasterthan a predetermined threshold, an event occurring periodically at apredetermined interval, an event occurring in response to the user'sspeech or text input, an event occurring as a consequence of change inoperation (e.g., motion and illuminance), and the like. The externalevent may include an event of receiving a message from outside(particularly, the message received from a target user device designatedin the currently running rule).

If the event is detected at step 1303, the control unit 170 checks thefunction to be executed which is specified in the execution rule at step1305. For example, the control unit 170 may check the function (orapplication) specified, in the currently running rule, as an action tobe execution in fulfillment of the corresponding event (condition).

The control unit 107 activates the checked function at step 1307 andexecutes the action by means of the activated function at step 1309. Forexample, if the action corresponding to the event (condition) specifiedin the execution rule is of controlling the audio volume of the userdevice 100, the control unit 170 controls to activate the audio volumecontrol function. If the action corresponding to the event (condition)specified in the execution rule is of sending current location of theuser device 100, the control unit 170 activates the location informationtransmission function (or application) such as GPS function (navigationfunction) and messaging function such that the user device 100 sends thetarget device the location information on the user device 100. Forexample, the control unit 170 may execute a function (application) or atleast two interoperable functions (applications) depending on the typeof the action to be executed.

The control unit 170 controls to feed back the associative informationas a result of the execution of the action (e.g. information resultingfrom the execution of the action) at step 1311. For example, the controlunit 170 may control to display a screen interface presenting audiovolume level to the user while adjusting the audio volume in response tothe user's manipulation. The control unit 170 also may control to outputthe screen interface and/or sound (audio effect) notifying of thetransmission of the location information of the user device 100 to thedesignated target device.

If no event (condition) is detected at step 1303, the control unit 170determines whether the current situation matches the rule releasecondition at step 1315. For example, the control unit 170 may monitorthe condition specified in the currently running rule and determinewhether the current situation matches the rule release conditionspecified in the rule by referencing the mapping table.

If the current situation does not match the rule release condition atstep 1315, the control unit 170 returns the procedure to continuemonitoring event at step 1301.

If the current situation matches the rule release condition at step1315, the control unit 170 releases the currently running rule at step1317 and feeds back the release information as a result of the releaseof the rule (e.g., information resulting from the release of the rule)at step 1319. For example, if the rule release condition is fulfilled,the control unit 170 may display a popup message prompting the user toterminate the currently running rule so as to maintain or release thecurrently running rule depending on the user's selection. The controlunit 170 may notify the user of the release of the currently runningrule in the form of at least one of audio, video, and tactile feedback.

FIGS. 31A to 31N are diagrams illustrating exemplary screens associatedwith an operation of generating a rule in a user device according to anexemplary embodiment of the present invention.

FIGS. 31A to 31N shows the procedure of defining a rule in such a waythat the control unit 170 (e.g., the rule configuration module 173)recognizes natural language-based text input of user scribing anddefines the rule (condition and action) in response to the recognizedtext input.

FIGS. 31A to 31N may correspond to the rule generation proceduredescribed above with reference to FIGS. 3A to 3K or FIGS. 4A to 4J. Forexample, FIG. 31A may correspond to FIG. 4A, FIG. 31C to FIG. 4B, FIG.31E to FIG. 4C, FIG. 31G to FIG. 4D, FIG. 31I to FIG. 4E, FIG. 31K toFIG. 4F, FIG. 31L to FIG. 4I, and FIG. 31N to FIG. 4J. In the followingdescription with reference to FIGS. 31A to 31N, the operations identicalwith or corresponding to those described with reference to FIGS. 4A to4J are omitted or mentioned briefly.

For example, FIGS. 31A to 31N show the operations of generating a rulein correspondence to the operations of FIGS. 4A to 4J according to anexemplary embodiment of the present invention. However, FIGS. 31A to 31Nare directed to the case in which the rule is generated based on theuser's text scribing input other than speech input. Unlike the speechinput-based rule generation procedure (FIGS. 4A to 4J), FIGS. 31A to 31Nshow the operations of generating a rule based on the user's textscribing input.

In the exemplary case of FIGS. 31A to 31N, the user input for generatinga rule is based on text and thus the user device 100 may provide thepopup message of “text input” for interaction with the user other than“speech input.” According to an exemplary embodiment of the presentinvention, the popup window 451 of FIG. 4A prompts the user to make aspeech input with a pictogram of speaking person and a text such as“please speak 000.” In FIGS. 31A, 31E, 31G, and 31L, however, the popupwindow 3151, 3155, 3157, and 3167 prompt the user to make a text inputwith a pictogram of pen-handwriting and a text such as “please write000.”

In the exemplary case of FIGS. 31A to 31N, the user input is made bywriting text rather than speaking the text. As shown in FIGS. 31B, 31D,31F, 31H, 31J, and 31M; the user device 100 may provide a writing windowto receive user's response, (e.g., text input) in response to therequest for providing information. According to an exemplary embodimentof the present invention, the user device 100 may display a guidance(query) such as “please write command” followed by displaying a writingwindow 3100 to receive the user's text (e.g., taxi) writing input.

According to various exemplary embodiments of the present invention, therule generation procedure may be progressed through interaction betweenthe user device 100 and the user, and the user may configure the rule bymaking natural language-based text input of conditions and actions ofthe rule according to the guidance of the user device 100. For example,the user device 100 may receive the natural language-based text inputmade by the user for configuring the conditions and rules constitutingthe rule and configure the corresponding rule according to the userinstruction input through the steps of FIGS. 31A to 31N.

FIGS. 32A to 32E are diagrams illustrating exemplary screens associatedwith an operation of executing a rule in a user device according to anexemplary embodiment of the present invention.

FIGS. 32A to 32E shows the exemplary screens of the user device 100 whenthe rule execution module 175 of the control unit 170 receives theuser's natural language-based text scribing and executes the rule inresponse to the user input, the condition check module 177 of thecontrol unit 170 monitors to detect the fulfillment of the conditionspecified in the rule, and the action execution module 179 executes atleast one action corresponding to the fulfilled condition.

FIGS. 32A to 32E may correspond to the rule execution proceduredescribed above with reference to FIGS. 5A to 5E. For example, FIG. 32Amay correspond to FIG. 5A, FIG. 32B to FIG. 5B, FIG. 32D to FIG. 5C, andFIG. 32E to FIG. 5D. In the following description with reference toFIGS. 32A to 32E, the operations identical with or corresponding tothose described with reference to FIGS. 5A to 5E are omitted ormentioned briefly.

For example, FIGS. 32A to 32E show the operations of executing a rule incorrespondence to the operations of FIGS. 5A to 5E according to anexemplary embodiment of the present invention. However, FIGS. 32A to 32Eare directed to the case in which the rule is generated based on theuser's text scribing input other than speech input. Unlike the speechinput-based rule execution procedure (FIGS. 5A to 5E), FIGS. 32A to 32Eshow the operations of executing a rule based on the user's textscribing input.

In the exemplary case of FIGS. 32A to 32E, the user input for executinga rule is based on text and thus the user device 100 may provide thepopup message of “text input” for interaction with the user other than“speech input.” According to an exemplary embodiment of the presentinvention, the popup window 551 of FIG. 5B prompts the user to make aspeech input with a pictogram of speaking person and a text such as“please speak 000.” In FIG. 32B, however, the popup window 3251 promptsthe user to make a text input with a pictogram of pen-handwriting and atext such as “please write 000.”

Referring to FIGS. 32A to 32E, the user input is made by writing textrather than speaking the text. As shown in FIG. 32C, the user device 100may provide a writing window to receive user's response, (e.g., textinput) in response to the request for providing information. Accordingto an exemplary embodiment of the present invention, the user device 100may display a guidance (query) such as “please write command” followedby displaying a writing window 3200 to receive the user's text (e.g.,subway) writing input.

According to various exemplary embodiments of the present invention, therule execution procedure may be progressed through interaction betweenthe user device 100 and the user, and the user may execute the rule bymaking natural language-based text input according to the guidance ofthe user device 100. For example, the user device 100 may receive thenatural language-based text input made by the user for executing therule according to the user instruction input through the steps of FIGS.32A to 32E.

According to various exemplary embodiments of the present invention, theuser may execute the rule using the widget 500. According to anexemplary embodiment of the present invention, the user may select aninstruction input region (or rule execution button) 510 of the widget500 to input a text of the configured rule (or command). Then thecontrol unit 170 feeds back the information on the action to be executedto the user in the form of a text popup or speech announcement (e.g.,Text-To-Speech (TTS)) along with the notification of the start of therule. The control unit 170 adds the executed rule to the executioninformation region 520 and displays a notification item notifying of theexistence of currently running rule at the indicator region.

As shown in FIG. 32A, the text input-based widget 500 may be provided inseparation from the speech input-based widget 500. According to anexemplary embodiment, the widget 500 of FIG. 5A may be provided with apictogram of speaking person in the instruction input region (or ruleexecution button) 510 to indicate that the speech recognition functionis running. The widget 500 of FIG. 32A may be provided with a pictogram(e.g., icon) of a pen-writing hand in the instruction input region (orrule execution button) 510 to indicate that the text recognitionfunction is running.

FIGS. 33A to 33D are diagrams illustrating exemplary screens associatedwith an operation of suspending a currently running rule in a userdevice according to an exemplary embodiment of the present invention.

Referring to FIGS. 33A to 33D, the control unit 170 recognizes a naturallanguage-based text scribing input made by the user and stops thecurrently running rule temporarily in response to the text input.

FIGS. 33A to 33D may correspond to the rule suspension proceduredescribed above with reference to FIGS. 16A to 16C. For example, FIG.33A may correspond to FIG. 16A, FIG. 33B to FIG. 16B, and FIG. 33D toFIG. 16C. In the following description with reference to FIGS. 33A to33D, the operations identical with or corresponding to those describedwith reference to FIGS. 16A to 16C are omitted or mentioned briefly.

For example, FIGS. 33A to 33D show the operations of suspending a rulein correspondence to the operations of FIGS. 16A to 16D according to anexemplary embodiment of the present invention. However, FIGS. 33A to 33Dare directed to the case in which the rule is generated based on theuser's text scribing input other than speech input. Unlike the speechinput-based rule execution procedure (FIGS. 16A to 16C), FIGS. 33A to33D show the operations of suspending a rule based on the user's textinput.

In the exemplary case of FIGS. 33A to 33D, the user input for suspendinga rule is based on text and thus the user device 100 may provide thepopup message of “text input” for interaction with the user other than“speech input.” According to an exemplary embodiment of the presentinvention, the popup window 3351 of FIG. 33B prompts the user to make atext input with a pictogram (e.g., icon) of pen-handwriting and a textsuch as “please write 000.”

In the exemplary case of FIGS. 33A to 33D, the user input is made byscribing a text other than speaking the text. As shown in FIG. 33C, theuser device 100 may provide a writing window 3300 to receive user'sresponse, (e.g., scribing input) in response to the request forproviding information. According to an exemplary embodiment of thepresent invention, the user device 100 may display a guidance (query)such as “please write command” followed by displaying a writing window3300 to receive the user's text (e.g., suspend taxi) writing input.

According to various exemplary embodiments of the present invention, therule suspension procedure may be progressed through interaction betweenthe user device 100 and the user, and the user may suspend the rule bymaking natural language-based text input according to the guidance ofthe user device 100. For example, the user device 100 may receive thenatural language-based text input made by the user for suspending therule according to the user instruction input through the steps of FIGS.33A to 33D.

According to various exemplary embodiments of the present invention, theuser may execute the rule using the widget 500. According to anexemplary embodiment of the present invention, the user may select aninstruction input region (or rule execution button) 510 of the widget500 to input a text instruction for suspending the currently runningrule. In this way, the user may stop the periodic operation of at leastone rule temporarily. For example, the control unit 170 may suspend thecorresponding rule in response to the user's text input such as “pleasestop 000 temporarily.”

FIGS. 34A to 34D are diagrams illustrating exemplary screens associatedwith an operation of terminating a currently running rule in a userdevice according to an exemplary embodiment of the present invention.

FIGS. 34A to 34D may correspond to the rule termination proceduredescribed above with reference to FIGS. 22A to 22C. For example, FIG.34A may correspond to FIG. 22A, FIG. 34B to FIG. 22B, and FIG. 34D toFIG. 34C. In the following description with reference to FIGS. 34A to34D, the operations identical with or corresponding to those describedwith reference to FIGS. 22A to 22C are omitted or mentioned briefly.

For example, FIGS. 34A to 34D show the operations of terminating a rulein correspondence to the operations of FIGS. 22A to 22D according to anexemplary embodiment of the present invention. However, FIGS. 34A to 34Dare directed to the case in which the rule is terminated based on theuser's text input other than speech input. Unlike the speech input-basedrule termination procedure (FIGS. 22A to 22C), FIGS. 34A to 34D show theoperations of terminating a rule based on the user's text scribinginput.

In the exemplary case of FIGS. 34A to 34D, the user input forterminating a rule is based on text and thus the user device 100 mayprovide the popup message of “text input” for interaction with the userother than “speech input.” According to an exemplary embodiment of thepresent invention, the popup window 3451 of FIG. 34B prompts the user tomake a text input with a pictogram of pen-handwriting and a text such as“please write 000.”

In the exemplary case of FIGS. 34A to 34D, the user input is made bywriting a text rather than speaking the text. As shown in FIG. 34C, theuser device 100 may provide a writing window 3400 to receive a user'sresponse, (e.g., text input) in response to the request for providinginformation. According to an exemplary embodiment of the presentinvention, the user device 100 may display a guidance (query) such as“please write command” followed by displaying a writing window 3400 toreceive the user's text (e.g., terminate subway) writing input as shownin FIG. 34C.

According to various exemplary embodiments of the present invention, therule termination procedure may be progressed through interaction betweenthe user device 100 and the user, and the user may terminate the rule bymaking natural language-based text input according to the guidance ofthe user device 100. For example, the user device 100 may receive thenatural language-based text input made by the user for terminating therule according to the user instruction input through the steps of FIGS.34A to 34D.

According to various exemplary embodiments of the present invention, theuser may execute the rule using the widget 500. According to anexemplary embodiment of the present invention, the user may select aninstruction input region (or rule execution button) 510 of the widget500 to input a text instruction for terminating the currently runningrule. Also, according to various exemplary embodiments of the presentinvention, the control unit may control such that when the correspondingrule is terminated according to the user's text input, the user device100 recovers the user device's configuration to the state before theexecution of the corresponding rule automatically.

The CAS provision method and apparatus according to exemplaryembodiments of the present invention is capable of configuring the rules(or conditions) of CAS diversely according to user's definition. Theuser device 100 recognizes the conditions specified in at lease on ruledefined by the user and executes at least one action in fulfillment ofthe corresponding conditions. The CAS provision method and apparatusaccording to exemplary embodiments of the present invention is capableof feeding back the internal and/or external context information to theuser as a result of the execution of the action.

The CAS provision method and apparatus according to exemplaryembodiments of the present invention is capable of defining the rules(or situation), instructions for executing the corresponding rules, andactions to be executed according to the rule through naturallanguage-based text or speech input using the user device 100.Accordingly, the CAS provision method and apparatus of the presentinvention allows the user to define various rules with user-specifiedconditions and action as well as the rules defined in the manufacturingstage of the user device 100. The CAS provision method and apparatusaccording to exemplary embodiments of the present invention is capableof defining the rules and instructions through natural language-basedtext or speech input and executing the rules in response to thenatural-language-based text or speech instruction or detection of themovement of the user device 100. Accordingly, exemplary embodiments ofthe present invention are capable of extending the range of CAS andimprove the user-specific usability.

The CAS provision method and apparatus according to exemplaryembodiments of the present invention of the present invention is capableof configuring plural conditions per rule and support multi-contextawareness scenarios corresponding to the plural conditions. Accordingly,the CAS provision method and apparatus according to exemplaryembodiments of the present invention is capable of configuring variousconditions according to the user preference and performing pluralactions simultaneously in correspondence to the multi-context awarenessscenarios. The CAS provision method and apparatus according to exemplaryembodiments of the present invention is capable of improving the contextawareness functionality by adopting the recommendation function as wellas context awareness, resulting in improvement of context recognitionaccuracy, as compared to the statistics-based context awarenesstechnology according to the related art.

The CAS provision method and apparatus according to exemplaryembodiments of the present invention is capable of optimizing the CASenvironment, resulting in improvement of user convenience and deviceusability, utilization, and competitiveness. The CAS provision methodand apparatus according to exemplary embodiments of the presentinvention is applicable to various types of CAS-capable devicesincluding cellular communication terminal, smartphone, tablet PC, PDA,and the like.

According to exemplary embodiments of the present invention, the modulesmay be implemented as any or combination of software, firmware,hardware, and/or any combination thereof. Some or all of the modules maybe implemented as an entity capable of executing the functions of theindividual modules identically. According to various exemplaryembodiments of the present invention, plural operations may be performedsequentially, repeatedly, or in parallel. Some of the operations may beomitted or replaced by others.

The above-described exemplary embodiments of the present invention canbe implemented in the form of computer-executable program commands andstored in a non-transient computer-readable storage medium. Thenon-transient computer-readable storage medium may store the programcommands, data files, and data structures in individual or combinedforms. The program commands recorded in the storage medium may bedesigned and implemented for various exemplary embodiments of thepresent invention or used by those skilled in the computer softwarefield.

The non-transient computer-readable storage medium includes magneticmedia such as a floppy disk and a magnetic tape, optical media includinga Compact Disc (CD) ROM and a Digital Video Disc (DVD) ROM, amagneto-optical media such as a floptical disk, and the hardware devicedesigned for storing and executing program commands such as ROM, RAM,and flash memory. The programs commands include the language codeexecutable by computers using the interpreter as well as the machinelanguage codes created by a compiler. The aforementioned hardware devicecan be implemented with one or more software modules for executing theoperations of the various exemplary embodiments of the presentinvention.

While the invention has been shown and described with reference tocertain exemplary embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims and their equivalents.

What is claimed is:
 1. A method for providing a context-aware serviceusing an electronic device, the method comprising at least: providing,for display on a touchscreen of the electronic device, a screen of acontext-aware service application being executed, the screen of thecontext-aware service application including a user selectable visualelement for adding a rule; based on receiving a first user input to theuser selectable visual element for adding the rule, providing, fordisplay on the touchscreen of the electronic device, an interactive userinterface for interacting with a user of the electronic device to addthe rule, the first user input being a touch input to the userselectable visual element for adding the rule; providing, for display onthe touchscreen of the electronic device, a first guidance by theinteractive user interface for adding at least one condition of therule, the first guidance guiding the user of the electronic device toenter text for a condition of the rule that is meetable based onreceiving speech that corresponds to the entered text; based onreceiving at least one second user input, the at least one second userinput being a text input, identifying a first condition of the rule, thefirst condition being meetable based on receiving speech thatcorresponds to the text input of the at least one second user input;providing, for display on the touchscreen of the electronic device, asecond guidance by the interactive user interface for adding at leastone action of the rule; based on receiving a third user input,identifying a first action of the rule based on the third user input,the first action of the rule corresponding to a first performableoperation; providing again, for display on the touchscreen of theelectronic device, the second guidance by the interactive user interfacefor adding at least one action of the rule; based on receiving a fourthuser input, identifying a second action of the rule based on the fourthuser input, the second action of the rule corresponding to a secondperformable operation; receiving a fifth user input for storing the ruleincluding the first condition of the rule, the first action of the rule,and the second action of the rule; based on receiving a sixth userinput, the sixth user input being a speech input for meeting the firstcondition, determining whether the first condition of the rule is metbased on whether the speech input of the sixth user input corresponds tothe text input of the at least one second user input; and based on adetermination that the first condition of the rule is met, performingboth the first action of the rule and the second action of the rule,wherein the first guidance by the interactive user interface for addingthe at least one condition of the rule that guides the user of theelectronic device to enter the text for the condition of the rule thatis meetable based on receiving the speech that corresponds to theentered text, and the second guidance by the interactive user interfacefor adding the at least one action of the rule, are provided, fordisplay on the touchscreen of the electronic device, at different times,and wherein upon the receiving of the second user input, the providing,for display on the touchscreen of the electronic device, of the firstguidance by the interactive user interface for adding the at least onecondition of the rule is discontinued.
 2. The method of claim 1, whereinthe determining of whether the first condition of the rule is met basedon whether the speech input of the sixth user input corresponds to thetext input of the at least one second user input comprises at least:determining whether the speech input of the sixth user input correspondsto the text input of the at least one second user input based on resultsof a speech recognition performed on the speech input of the sixth userinput.
 3. The method of claim 1, further comprising at least: providingagain, for display on the touchscreen of the electronic device, thefirst guidance by the interactive user interface for adding at least onecondition of the rule; and based on receiving a seventh user input,identifying a second condition of the rule based on the seventh userinput, wherein the second condition of the rule is based on at least oneof location, schedule, time, an event of the electronic device, or oneor more spoken words or sounds.
 4. The method of claim 1, wherein thefirst action and the second action are each at least one of: sending amessage, changing a device setting, music playback, providing weatherinformation, providing information from an external server, providinginformation to the external server, or providing control of an externaldevice.
 5. The method of claim 1, further comprising at least: based onreceiving a user selection of a button for inputting the sixth userinput, entering a mode for receiving the speech input of the sixth userinput.
 6. The method of claim 1, wherein the screen of the context-awareservice application including the user selectable visual element foradding the rule includes one or more delineated areas that eachcorrespond to a saved rule.
 7. The method of claim 6, wherein eachdelineated area of the one or more delineated areas includes at leastone of an icon, a textual description, and another user selectablevisual element, and wherein the other user selectable visual element isfor providing, for display on a touchscreen of the electronic device,information regarding a corresponding rule based on the other userselectable visual element being selected.
 8. The method of claim 1,further comprising at least: based on receiving the text input of the atleast one second user input, providing, for display on a touchscreen ofthe electronic device, the text input of the at least one second userinput.
 9. The method of claim 1, further comprising at least: based on auser selection of another user selectable visual element, the other userselectable visual element being for confirming the rule, confirming therule including the first condition of the rule, the first action of therule, and the second action of the rule.
 10. The method of claim 1,wherein data for the context-aware service is communicated with anexternal server.
 11. A non-transitory computer readable storage mediumwith instructions stored thereon to provide a context-aware serviceusing an electronic device, the instructions executable by a processorof the electronic device to cause the processor to at least: control toprovide, for display on a touchscreen of the electronic device, a screenof a context-aware service application being executed, the screen of thecontext-aware service application including a user selectable visualelement for adding a rule; based on receiving a first user input to theuser selectable visual element for adding the rule, control to provide,for display on the touchscreen of the electronic device, an interactiveuser interface for interacting with a user of the electronic device toadd the rule, the first user input being a touch input to the userselectable visual element for adding the rule; control to provide, fordisplay on the touchscreen of the electronic device, a first guidance bythe interactive user interface for adding at least one condition of therule, the first guidance guiding the user of the electronic device toenter text for a condition of the rule that is meetable based onreceiving speech that corresponds to the entered text; based onreceiving at least one second user input, the at least one second userinput being a text input, control to identify a first condition of therule, the first condition being meetable based on receiving speech thatcorresponds to the text input of the at least one second user input;control to provide, for display on the touchscreen of the electronicdevice, a second guidance by the interactive user interface for addingat least one action of the rule; based on receiving a third user input,control to identify a first action of the rule based on the third userinput, the first action of the rule corresponding to a first performableoperation; control to provide again, for display on the touchscreen ofthe electronic device, the second guidance by the interactive userinterface for adding at least one action of the rule; based on receivinga fourth user input, control to identify a second action of the rulebased on the fourth user input, the second action of the rulecorresponding to a second performable operation; control to receive afifth user input for storing the rule including the first condition ofthe rule, the first action of the rule, and the second action of therule; based on receiving a sixth user input, the sixth user input beinga speech input for meeting the first condition, control to determinewhether the first condition of the rule is met based on whether thespeech input of the sixth user input corresponds to the text input ofthe at least one second user input; and based on a determination thatthe first condition of the rule is met, control to perform both thefirst action of the rule and the second action of the rule, wherein thefirst guidance by the interactive user interface for adding the at leastone condition of the rule that guides the user of the electronic deviceto enter the text for the condition of the rule that is meetable basedon receiving the speech that corresponds to the entered text, and thesecond guidance by the interactive user interface for adding the atleast one action of the rule, are controlled to be provided, for displayon the touchscreen of the electronic device, at different times, andwherein upon the receiving of the second user input, the control toprovide, for display on the touchscreen of the electronic device, thefirst guidance by the interactive user interface for adding the at leastone condition of the rule is discontinued.
 12. The non-transitorycomputer readable storage medium of claim 11, wherein the control todetermine whether the first condition of the rule is met based onwhether the speech input of the sixth user input corresponds to the textinput of the at least one second user input comprises at least: controlto determine whether the speech input of the sixth user inputcorresponds to the text input of the at least one second user inputbased on results of a speech recognition performed on the speech inputof the sixth user input.
 13. The non-transitory computer readablestorage medium of claim 11, wherein the instructions cause the processorfurther to at least: control to provide again, for display on thetouchscreen of the electronic device, the first guidance by theinteractive user interface for adding at least one condition of therule; and based on receiving a seventh user input, control to identify asecond condition of the rule based on the seventh user input, whereinthe second condition of the rule is based on at least one of location,schedule, time, an event of the electronic device, or one or more spokenwords or sounds.
 14. The non-transitory computer readable storage mediumof claim 11, wherein the first action and the second action are each atleast one of: sending a message, changing a device setting, musicplayback, providing weather information, providing information from anexternal server, providing information to the external server, orproviding control of an external device.
 15. The non-transitory computerreadable storage medium of claim 11, wherein the instructions cause theprocessor further to at least: based on receiving a user selection of abutton for inputting the sixth user input, control to enter a mode forreceiving the speech input of the sixth user input.
 16. Thenon-transitory computer readable storage medium of claim 11, wherein thescreen of the context-aware service application including the userselectable visual element for adding the rule includes one or moredelineated areas that each correspond to a saved rule.
 17. Thenon-transitory computer readable storage medium of claim 16, whereineach delineated area of the one or more delineated areas includes atleast one of an icon, a textual description, and another user selectablevisual element, and wherein the other user selectable visual element isfor providing, for display on a touchscreen of the electronic device,information regarding a corresponding rule based on the other userselectable visual element being selected.
 18. The non-transitorycomputer readable storage medium of claim 11, wherein the instructionscause the processor further to at least: based on receiving the textinput of the at least one second user input, control to provide, fordisplay on a touchscreen of the electronic device, the text input of theat least one second user input.
 19. The non-transitory computer readablestorage medium of claim 11, wherein the instructions cause the processorfurther to at least: based on a user selection of another userselectable visual element, the other user selectable visual elementbeing for confirming the rule, controlling to confirm the rule includingthe first condition of the rule, the first action of the rule, and thesecond action of the rule.
 20. The non-transitory computer readablestorage medium of claim 11, wherein data for the context-aware serviceis communicated with an external server.
 21. An electronic device usedto provide a context-aware service, the electronic device comprising: amemory storing instructions; and a processor configured to execute thestored instructions to at least: control to provide, for display on atouchscreen of the electronic device, a screen of a context-awareservice application being executed, the screen of the context-awareservice application including a user selectable visual element foradding a rule; based on receiving a first user input to the userselectable visual element for adding the rule, control to provide, fordisplay on the touchscreen of the electronic device, an interactive userinterface for interacting with a user of the electronic device to addthe rule, the first user input being a touch input to the userselectable visual element for adding the rule; control to provide, fordisplay on the touchscreen of the electronic device, a first guidance bythe interactive user interface for adding at least one condition of therule, the first guidance guiding the user of the electronic device toenter text for a condition of the rule that is meetable based onreceiving speech that corresponds to the entered text; based onreceiving at least one second user input, the at least one second userinput being a text input, control to identify a first condition of therule, the first condition being meetable based on receiving speech thatcorresponds to the text input of the at least one second user input;control to provide, for display on the touchscreen of the electronicdevice, a second guidance by the interactive user interface for addingat least one action of the rule; based on receiving a third user input,control to identify a first action of the rule based on the third userinput, the first action of the rule corresponding to a first performableoperation; control to provide again, for display on the touchscreen ofthe electronic device, the second guidance by the interactive userinterface for adding at least one action of the rule; based on receivinga fourth user input, control to identify a second action of the rulebased on the fourth user input, the second action of the rulecorresponding to a second performable operation; control to receive afifth user input for storing the rule including the first condition ofthe rule, the first action of the rule, and the second action of therule; based on receiving a sixth user input, the sixth user input beinga speech input for meeting the first condition, control to determinewhether the first condition of the rule is met based on whether thespeech input of the sixth user input corresponds to the text input ofthe at least one second user input; and based on a determination thatthe first condition of the rule is met, control to perform both thefirst action of the rule and the second action of the rule, wherein thefirst guidance by the interactive user interface for adding the at leastone condition of the rule that guides the user of the electronic deviceto enter the text for the condition of the rule that is meetable basedon receiving the speech that corresponds to the entered text, and thesecond guidance by the interactive user interface for adding the atleast one action of the rule, are controlled to be provided, for displayon the touchscreen of the electronic device, at different times, andwherein upon the receiving of the second user input, the control toprovide, for display on the touchscreen of the electronic device, thefirst guidance by the interactive user interface for adding the at leastone condition of the rule is discontinued.
 22. The electronic device ofclaim 21, wherein the control to determine whether the first conditionof the rule is met based on whether the speech input of the sixth userinput corresponds to the text input of the at least one second userinput comprises at least: control to determine whether the speech inputof the sixth user input corresponds to the text input of the at leastone second user input based on results of a speech recognition performedon the speech input of the sixth user input.
 23. The electronic deviceof claim 21, wherein the processor is configured to execute the storedinstructions further to at least: control to provide again, for displayon the touchscreen of the electronic device, the first guidance by theinteractive user interface for adding at least one condition of therule; and based on receiving a seventh user input, control to identify asecond condition of the rule based on the seventh user input, whereinthe second condition of the rule is based on at least one of location,schedule, time, an event of the electronic device, or one or more spokenwords or sounds.
 24. The electronic device of claim 21, wherein thefirst action and the second action are each at least one of: sending amessage, changing a device setting, music playback, providing weatherinformation, providing information from an external server, providinginformation to the external server, or providing control of an externaldevice.
 25. The electronic device of claim 21, wherein the processor isconfigured to execute the stored instructions further to at least: basedon receiving a user selection of a button for inputting the sixth userinput, control to enter a mode for receiving the speech input of thesixth user input.
 26. The electronic device of claim 21, wherein thescreen of the context-aware service application including the userselectable visual element for adding the rule includes one or moredelineated areas that each correspond to a saved rule.
 27. Theelectronic device of claim 26, wherein each delineated area of the oneor more delineated areas includes at least one of an icon, a textualdescription, and another user selectable visual element, and wherein theother user selectable visual element is for providing, for display on atouchscreen of the electronic device, information regarding acorresponding rule based on the other user selectable visual elementbeing selected.
 28. The electronic device of claim 21, wherein theprocessor is configured to execute the stored instructions further to atleast: based on receiving the text input of the at least one second userinput, control to provide, for display on a touchscreen of theelectronic device, the text input of the at least one second user input.29. The electronic device of claim 21, wherein the processor isconfigured to execute the stored instructions further to at least: basedon a user selection of another user selectable visual element, the otheruser selectable visual element being for confirming the rule,controlling to confirm the rule including the first condition of therule, the first action of the rule, and the second action of the rule.30. The electronic device of claim 21, wherein data for thecontext-aware service is communicated with an external server.